Image analysis method, device, computer program, and method for generating deep learning algorithm

Abstract

To provide an image analysis method capable of identifying each form in a plurality of cells included in an analysis image with accuracy.SOLUTION: There is provided the image analysis method for analyzing a form of cells, using a deep learning algorithm comprising a neural network structure, the method is configured so that, analysis data including information related to cells which are analysis objects is input to the deep learning algorithm (60, 61), then the deep learning algorithm (60, 61) calculates a probability at which the cells being analysis objects belong to each form of form classifications for a plurality of cells belonging to a prescribed cell group, for analyzing the image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to image analysis methods, devices, computer programs, and methods for generating deep learning algorithms for analyzing cell morphology. [Background technology]

[0002] Patent Document 1 discloses a cell classification system for processing microscopic images. In the cell classification system, a model trained using a machine training technique associates pixels in an acquired image with one or more of a cell, a cell edge, a background, and the like. The machine training technique uses a random forest decision tree technique. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special table number 2016-534709 Summary of the Invention [Problem to be solved by the invention]

[0004] In cytological testing, examiners usually observe cells under a microscope and morphologically identify the cell type and characteristics. However, because cells of the same lineage have similar morphologies, in order to be able to morphologically identify cells, it is necessary to observe many cell specimens and improve identification skills. Skill is particularly required to identify abnormal cells that appear when a patient is infected with a disease. For example, when the frequency of abnormal cells is low, such as in the early stages of myelodysplastic syndrome, there is a risk that examiners with insufficient skills will not notice abnormal cells.

[0005] In addition, there is a limit to the number of specimens that an examiner can observe in a day, and observing more than 100 specimens in a day is also a burden for the examiner.

[0006] It is possible to increase the number of cell tests processed by using an automated blood cell sorter that employs a flow method. However, the information that can be obtained from an automated blood cell sorter using the flow method is limited, and it is difficult to distinguish blood cells that appear infrequently, such as blast cells, promyelocytes, and giant platelets.

[0007] As in the method described in Patent Document 1, a method of identifying cells using a machine training technique (also called machine learning) is also known. However, the user needs to create training data for training the machine learning model, and an enormous amount of effort is required to generate the model. In addition, since the user creates the training data, the number of training data that can be created is limited, and there are problems with the accuracy of analysis and generalization performance of the machine learning model.

[0008] In addition, the method described in Patent Document 1 is a method for distinguishing between cellular and non-cellular parts in a microscopic image, and therefore cannot distinguish the type of each cell or the abnormal findings it contains.

[0009] An object of the present disclosure is to provide an image analysis method for identifying the morphology of each of a plurality of cells contained in an analysis image with higher accuracy. [Means for solving the problem]

[0010] An embodiment of the present disclosure relates to an image analysis method for analyzing cell morphology using a deep learning algorithm (50, 51) having a neural network structure. In the image analysis method, analysis data (80) including information on a cell to be analyzed is input to a deep learning algorithm (60, 61) having a neural network structure, and the deep learning algorithm calculates the probability that the cell to be analyzed belongs to each of the morphological classifications of a plurality of cells belonging to the predetermined cell group, and analyzes the image. With this embodiment, even if an examiner does not perform microscopic observation, it is possible to calculate the probability that the cell to be analyzed belongs to each of the morphological classifications of a plurality of cells belonging to the predetermined cell group and analyze the image.

[0011] Preferably, the image analysis method identifies the morphological classification of the cell to be analyzed as belonging to a plurality of cell morphological classifications of cells belonging to a predetermined cell group based on the calculated probability. According to this embodiment, it is possible to identify the morphological classification of the cell to be analyzed as belonging to a plurality of cell morphological classifications without the need for an examiner to observe the cell under a microscope.

[0012] Preferably, the predetermined cell group is a group of blood cells. According to this embodiment, an examiner can classify the morphology of blood cells without observing them using a microscope.

[0013] The predetermined cell group is preferably a group of cells belonging to a predetermined cell lineage. More preferably, the predetermined cell lineage is a hematopoietic lineage. According to this embodiment, the examiner can morphologically classify cells belonging to the same cell lineage without performing microscopic observation.

[0014] Preferably, the morphological classification is a distinction between types of cells to be analyzed. More preferably, the morphological classification includes neutrophils, including segmented neutrophils and band neutrophils, metamyelocytes, myelocytes, promyelocytes, blasts, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts, giant platelets, platelet clumps, and megakaryocytes. According to this embodiment, cells of the same lineage having similar morphologies can be distinguished.

[0015] Preferably, the morphological classification is identification of abnormal findings of the cells to be analyzed, and more preferably, the morphological classification includes at least one cell selected from the group consisting of morphological nuclear abnormality, presence of vacuoles, granular morphological abnormality, abnormal granular distribution, presence of abnormal granules, abnormal cell size, presence of inclusion bodies, and bare nuclei. According to this embodiment, even cells exhibiting abnormal findings can be identified.

[0016] In the embodiment, the data on the cell morphology is data on the cell type in a morphological classification and data on the cell features in a morphological classification. This embodiment makes it possible to output the morphological cell type and the cell features.

[0017] In the embodiment, the deep learning algorithm preferably includes a first algorithm for calculating the probability that the cell to be analyzed belongs to each of the first morphological classifications of a plurality of cells belonging to a predetermined cell group, and a first algorithm for calculating the probability that the cell to be analyzed belongs to each of the second morphological classifications of a plurality of cells belonging to a predetermined cell group. For example, the first morphological classification is the type of the cell to be analyzed, and the second morphological classification is an abnormal finding of the cell to be analyzed. In this way, the accuracy of identifying cells having similar morphologies can be further improved.

[0018] In the embodiment, the analysis data (80) is image data of stained blood cells. More preferably, the stain is selected from Wright's stain, Giemsa stain, Wright-Giemsa stain, and May-Giemsa stain. In this way, it is possible to perform identification similar to that performed by conventional observation under a microscope.

[0019] The analysis data (80) and the training data (75) include information on the luminance of the analysis target image and the training image, and information on at least two types of hues, thereby improving the classification accuracy.

[0020] Another embodiment of the present disclosure relates to an image analysis device (200) that analyzes cell morphology using a deep learning algorithm having a neural network structure. The image analysis device (200) includes a processing unit (10) that inputs analysis data (80) including information on a cell to be analyzed to a classifier including the deep learning algorithm (60, 61), and calculates the probability that the cell to be analyzed belongs to each of a plurality of morphological classifications of cells belonging to the predetermined cell group by the deep learning algorithm (60, 61) to analyze an image. Preferably, the morphological classification is a classification of the type of the cell to be analyzed. Also, preferably, the morphological classification is a classification of abnormal findings of the cell to be analyzed.

[0021] Another embodiment of the present disclosure relates to a computer program for image analysis that analyzes cell morphology using a deep learning algorithm (60, 61) having a neural network structure. The computer program causes a computer (200) to execute a process of analyzing an image by calculating a probability that the cell to be analyzed belongs to each of a plurality of morphological classifications of cells belonging to the predetermined cell group using analysis data (83) including information on the cell to be analyzed, using the deep learning algorithm (60, 61). Preferably, the morphological classification is identification of the type of the cell to be analyzed. Also, preferably, the morphological classification is identification of abnormal findings of the cell to be analyzed.

[0022] Another embodiment of the present disclosure relates to a method for generating a trained deep learning algorithm (60, 61). In this embodiment, training data including information about cells is input to an input layer (50a, 50b) of a neural network (50, 51), and label values ​​associated with morphological classifications of a plurality of cells belonging to a predetermined cell group are input as an output layer (51a, 51b). Preferably, the morphological classification is identification of the type of cells to be analyzed. Also, preferably, the morphological classification is identification of abnormal findings of the cells to be analyzed.

[0023] By using the image analysis device (200) and trained deep learning algorithms (60, 61), morphological cell types and cell features can be identified without being affected by the skill of the examiner. Effect of the Invention

[0024] The morphology of each of the multiple cells contained in the analysis image can be identified, making it possible to perform cell inspection that is not affected by the skill of the inspector. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram showing an overview of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram showing an example of a procedure for generating training data and a procedure for training a first deep learning algorithm and a second deep learning algorithm. [Diagram 3] Examples of label values ​​are as follows: [Figure 4] FIG. 1 is a schematic diagram showing an example of a procedure for generating analysis data and a procedure for identifying cells using a deep learning algorithm. [Diagram 5] 1 is a diagram illustrating an outline of a configuration example of an image analysis system 1. FIG. [Figure 6] 2 is a block diagram showing an example of a hardware configuration of a vendor-side device 100. FIG. [Figure 7] FIG. 2 is a block diagram showing an example of a hardware configuration of a user device 200. [Figure 8] A block diagram for explaining an example of the functions of deep learning device 100A. [Figure 9] 1 is a flowchart showing an example of the flow of a deep learning process. [Figure 10] FIG. 1 is a schematic diagram for explaining a neural network. [Figure 11] FIG. 2 is a block diagram for explaining an example of the functions of the image analysis device 200A. [Figure 12] 13 is a flowchart showing an example of the flow of an image analysis process. [Figure 13]FIG. 2 is a diagram illustrating an outline of a configuration example of an image analysis system 2. [Figure 14] FIG. 2 is a block diagram for explaining an example of the functions of an integrated image analyzing device 200B. [Figure 15] FIG. 2 is a diagram illustrating an outline of a configuration example of an image analysis system 3. [Figure 16] FIG. 2 is a block diagram for explaining an example of the functions of an integrated image analyzing device 100B. [Figure 17] 4 shows the results of cell type identification using a deep learning algorithm. [Figure 18] The results of identifying cell features using a deep learning algorithm are shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an overview and embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and drawings, the same reference numerals indicate the same or similar components, and therefore, the description of the same or similar components will be omitted.

[0027] [1. Image analysis method] A first embodiment of the present disclosure relates to an image analysis method for analyzing cell morphology. The image analysis method inputs analysis data including information on a cell to be analyzed to a classifier including a deep learning algorithm having a neural network structure. The classifier calculates a probability that the cell to be analyzed belongs to each of the morphological classifications of a plurality of cells belonging to the predetermined cell group. Preferably, the image analysis method further includes identifying the cell to be analyzed as one of the morphological classifications of a plurality of cells belonging to the predetermined cell group based on the probability.

[0028] In the first embodiment, the cells to be analyzed belong to a predetermined cell group. The predetermined cell group is a cell group that constitutes each organ in the body of a mammal or bird. The predetermined cell group normally includes a plurality of cell types that are morphologically classified by histological or cytological microscopic observation. Morphological classification (also called "morphological classification") includes classification of cell types and classification of morphological characteristics of cells. Preferably, the cells to be analyzed are a cell group that belongs to a predetermined cell lineage that belongs to the predetermined cell group. The predetermined cell lineage is a cell group that belongs to the same lineage differentiated from a certain type of tissue stem cell. The predetermined cell lineage is preferably the hematopoietic system, and more preferably cells in the blood (also called "blood cells").

[0029] In the conventional method, hematopoietic cells are morphologically classified by a human observing a specimen stained for bright field in the bright field of a microscope. The stain is preferably selected from Wright's stain, Giemsa stain, Wright-Giemsa stain, and May-Giemsa stain. May-Giemsa stain is more preferable. The specimen is not limited as long as the morphology of each cell belonging to a predetermined cell group can be individually observed. Examples include smears and imprint specimens. A smear of peripheral blood or bone marrow is preferable, and a smear of peripheral blood is more preferable.

[0030] Morphologically, blood cells include cell types such as neutrophils, including segmented and band-shaped neutrophils, metamyelocytes, myelocytes, promyelocytes, blasts, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts (nucleated red blood cells, including proerythroblasts, basophilic erythroblasts, polychromatic erythroblasts, normochromatic erythroblasts, promegaloblasts, basophilic megaloblasts, polychromatic megaloblasts, and normochromatic megaloblasts), giant platelets, platelet aggregates, and megakaryocytes (nucleated megakaryocytes, including micromegakaryocytes).

[0031] The predetermined cell group may include abnormal cells exhibiting morphological abnormalities in addition to normal cells. The abnormalities appear as characteristics of cells classified morphologically. Examples of abnormal cells are cells that appear when a patient is affected by a predetermined disease, such as tumor cells. In the case of the hematopoietic system, the predetermined disease is a disease selected from the group consisting of myelodysplastic syndrome, leukemia (including acute myeloblastic leukemia, acute myeloblastic leukemia, acute promyelocytic leukemia, acute myelomonocytic leukemia, acute monocytic leukemia, erythroleukemia, acute megakaryoblastic leukemia, acute myelogenous leukemia, acute lymphocytic leukemia, lymphoblastic leukemia, chronic myelogenous leukemia, and chronic lymphocytic leukemia, etc.), malignant lymphoma (Hodgkin's lymphoma, non-Hodgkin's lymphoma, etc.), and multiple myeloma. In the case of the hematopoietic system, the abnormal finding is a cell having at least one morphological characteristic selected from the group consisting of morphological nuclear abnormality, presence of vacuoles, granular morphological abnormality, granular distribution abnormality, presence of abnormal granules, abnormal cell size, presence of inclusion bodies, and bare nucleus.

[0032] Morphonuclear abnormalities include small nuclei, large nuclei, hypersegmented nuclei, nuclei that are not segmented when they should normally be segmented (including pseudo-Pelger's nuclear abnormality), vacuoles, enlarged nucleoli, cleaved nuclei, and the presence of two nuclei in a cell when normally one nucleus should be present.

[0033] Abnormalities in the overall morphology of the cells include those having vacuoles in the cytoplasm (also known as vacuolar degeneration), those having morphological abnormalities in granules such as azurophilic granules, neutrophilic granules, acidophilic granules, basophilic granules, etc., those having abnormalities in the distribution of the above granules (excess, or reduction or disappearance), those having abnormal granules (e.g., toxic granules, etc.), those having abnormal cell size (larger or smaller than normal), those having inclusion bodies (Dahley bodies, Auer bodies, etc.), and those having bare nuclei.

[0034] <Outline of image analysis method> Using Figure 1, we will outline the image analysis method. The classifier used in the image analysis method includes a plurality of deep learning algorithms (sometimes simply referred to as "algorithms") having a neural network structure. Preferably, the classifier includes a first deep learning algorithm (50) and a second deep learning algorithm (51), in which the first deep learning algorithm (50) extracts a feature amount of a cell, and the second deep learning algorithm (51) identifies the cell to be analyzed based on the feature amount extracted by the first deep learning algorithm. More preferably, the classifier may include, in addition to the second deep learning algorithm, a plurality of types of deep learning algorithms (may be numbered as second, third, fourth, fifth, ... i-th) trained according to the purpose of identification downstream of the first deep learning algorithm as shown in FIG. 1. For example, the second deep learning algorithm identifies the type of cell based on the above-mentioned morphological classification. Also, for example, the third deep learning algorithm identifies the features of the cell based on the above-mentioned morphological classification for each feature. Preferably, the first deep learning algorithm is a convolution-connected neural network, and the second or subsequent deep learning algorithms downstream of the first deep learning algorithm are fully-connected neural networks.

[0035] Next, a method for generating training data 75 and a method for analyzing images will be described using the examples shown in Figures 2 to 4. For convenience of explanation, the following description will be given using a first deep learning algorithm and a second deep learning algorithm.

[0036] <Generating training data> The training images 70 used to train the deep learning algorithm are images of cells whose cell types and cell characteristics are known based on morphological classification corresponding to the cells to be analyzed. The specimen for capturing the training images 70 is preferably prepared from a sample containing cells of the same type as the cells to be analyzed, using the same specimen preparation method and staining method as the specimen containing the cells to be analyzed. In addition, the training images 70 are preferably captured under the same conditions as the imaging conditions of the cells to be analyzed.

[0037] The training image 70 can be acquired in advance for each cell using an imaging device such as a known optical microscope or a virtual slide scanner. In the example shown in FIG. 2, the training image 70 is a raw image captured with 360 pixels x 365 pixels using a Sysmex DI-60, and reduced to 255 pixels x 255 pixels, but this reduction is not essential. The number of pixels in the training image 70 is not limited as long as analysis is possible, but it is preferable that one side exceeds 100 pixels. In the example shown in FIG. 2, red blood cells are present around a neutrophil, but the image may be trimmed so that only the target cell is included. At least one cell to be trained is included in one image (red blood cells and normal-sized platelets may be included), and if the pixels corresponding to the cell to be trained are present in about 1 / 9 or more of the total pixels of the image, it can be used as the training image 70.

[0038] For example, in this embodiment, the imaging device preferably captures images in RGB color and CMY color. The color image preferably represents the shade or brightness of each primary color, such as red, green, and blue, or cyan, magenta, and yellow, as a 24-bit value (8 bits x 3 colors). The training image 70 may include at least one hue and the shade or brightness of that hue, but more preferably includes at least two hues and the shade or brightness of each hue. Information including a hue and the shade or brightness of that hue is also called a color tone.

[0039] Next, the color tone information of each pixel is converted, for example, from RGB color to a format including brightness information and hue information. As a format including brightness information and hue information, YUV (YCbCr, YPbPr, YIQ, etc.) can be given. Here, conversion to YCbCr format is explained as an example. Here, since the training image is RGB color, it is converted to brightness 72Y, a first hue (for example, blue-based) 72Cb, and a second hue (for example, red-based) 72Cr. Conversion from RGB to YCbCr can be performed by a known method. For example, conversion from RGB to YCbCr can be performed according to the international standard ITU-R BT.601. The converted brightness 72Y, first hue 72Cb, and second hue 72Cr can be expressed as a matrix of gradation values ​​as shown in FIG. 2 (hereinafter, also referred to as color tone matrix 72y, 72cb, 72cr). The luminance 72Y, the first hue 72Cb, and the second hue 72Cr are each expressed in 256 gradations ranging from 0 to 255. Here, instead of the luminance 72Y, the first hue 72Cb, and the second hue 72Cr, the training images may be converted using the three primary colors of red R, green G, and blue B, or the three primary colors of cyan C, magenta M, and yellow Y.

[0040] Next, based on the color tone matrices 72y, 72cb, 72cr, color tone vector data 74 is generated for each pixel by combining three gradation values, luminance 72y, first hue 72cb, and second hue 72cr.

[0041] Next, for example, since the training image 70 in Fig. 2 is a segmented neutrophil, a label value 77 of "1" indicating that the training image 70 is a segmented neutrophil is assigned to each color tone vector data 74 generated from the training image 70 in Fig. 2, resulting in training data 75. For convenience, the training data 75 is shown as 3 pixels x 3 pixels in Fig. 2, but in reality there is as much color tone vector data as there are pixels when the training data 70 is captured. 3 shows an example of the label value 77. Different label values ​​77 are assigned depending on the type of cell and the presence or absence of characteristics of each cell.

[0042] <Overview of deep learning> An overview of neural network training will be described with reference to FIG. 2 as an example. Both the first neural network 50 and the second neural network 51 are preferably convolutional neural networks. The number of nodes in the input layer 50a of the first neural network 50 corresponds to the product of the number of pixels in the input training data 75 and the number of luminances and hues contained in the image (for example, in the above example, there are three: luminance 72y, first hue 72cb, and second hue 72cr). The hue vector data 74 is input as a set 76 to the input layer 50a of the first neural network 50. The first neural network 50 is trained using the label values ​​77 of each pixel of the training data 75 as the output layer 50b of the first neural network.

[0043] The first neural network 50 extracts features of the above-mentioned morphological cell types and cell features based on the training data 75. The output layer 50b of the first neural network outputs results reflecting these features. The input layer 51a of the second neural network 51 receives the results output from the softmax function of the first neural network 50. In addition, since cells belonging to a certain cell lineage have similar cell morphologies, the deep learning algorithm 51 having the second neural network 51 is further trained to specialize in identifying specific morphological cell types and specific cell features. Therefore, the label values ​​77 of the training data 75 are also input to the output layer of the second neural network. Reference numerals 50c and 51c in FIG. 2 denote intermediate layers.

[0044] A first deep learning algorithm 60 having the thus trained first neural network 60 and a second deep learning algorithm 61 having a second neural network 61 are combined and used as an identifier for identifying which of a number of cell types belonging to a specified cell group and morphologically classified the cell to which the cell being analyzed corresponds.

[0045] <Image analysis method> FIG. 4 shows an example of an image analysis method. In the image analysis method, analysis data 81 is generated from an analysis image 78 obtained by capturing an image of a cell to be analyzed. The analysis image 78 is an image obtained by capturing an image of a cell to be analyzed. The analysis image 78 can be acquired using an imaging device such as a known optical microscope or a virtual slide scanner. In the example shown in FIG. 4, the analysis image 78 is obtained by reducing a raw image captured at 360 pixels x 365 pixels to 255 pixels x 255 pixels using a Sysmex DI-60, as in the training image 70, but this reduction is not essential. The number of pixels in the training image 70 is not limited as long as the analysis can be performed, but it is preferable that one side exceeds 100 pixels. In the example shown in FIG. 4, red blood cells are present around a segmented neutrophil, but the image may be trimmed so that only the target cell is included. An image can be used as an analysis image 78 if it contains at least one cell to be trained (red blood cells and normal-sized platelets may also be included) and the pixels corresponding to the cell to be trained account for at least 1 / 9 of the total pixels of the image.

[0046] For example, in this embodiment, the imaging device preferably captures images in RGB color and CMY color, etc. The color image preferably represents the shading or brightness of each primary color, such as red, green, and blue, or cyan, magenta, and yellow, as a 24-bit value (8 bits x 3 colors). The analysis image 78 may include at least one hue and the shading or brightness of that hue, but more preferably includes at least two hues and the shading or brightness of each hue. Information including the hue and the shading or brightness of that hue is also called color tone.

[0047] For example, the RGB color is converted into a format including luminance information and hue information. As a format including luminance information and hue information, YUV (YCbCr, YPbPr, YIQ, etc.) can be mentioned. Here, conversion to YCbCr format is explained as an example. Here, since the training image is RGB color, it is converted into luminance 79Y, a first hue (e.g., blue-based) 79Cb, and a second hue (e.g., red-based) 79Cr. Conversion from RGB to YCbCr can be performed by a known method. For example, conversion from RGB to YCbCr can be performed according to the international standard ITU-R BT.601. The converted luminance 79Y, first hue 79Cb, and second hue 79Cr can be expressed as a matrix of gradation values ​​as shown in FIG. 2 (hereinafter, also referred to as color tone matrix 79y, 79cb, 79cr). The luminance 72Y, the first hue 72Cb, and the second hue 72Cr are each expressed in 256 gradations ranging from 0 to 255. Here, instead of the luminance 79Y, the first hue 79Cb, and the second hue 79Cr, the training images may be converted using the three primary colors of red R, green G, and blue B, or the three primary colors of cyan C, magenta M, and yellow Y.

[0048] Next, based on the color tone matrices 79y, 79cb, 79cr, color tone vector data 80 is generated for each pixel by combining three gradation values, luminance 79y, first hue 79cb, and second hue 79cr. A collection of color tone vector data 80 generated from one analysis image 78 is generated as analysis data 81.

[0049] It is preferable that the analysis data 81 and the training data 75 are generated under the same imaging conditions and the same conditions for generating vector data to be input from each image to the neural network.

[0050] The analysis data 81 is input to the input layer 60a of the first neural network 60 constituting the trained first deep learning algorithm 60. The first deep learning algorithm extracts features from the analysis data 81, and outputs the results from the output layer 60c of the first neural network 60. The values ​​output from the output layer 60c are the probabilities that the cell to be analyzed, which is included in the analysis image, belongs to each of the morphological cell classifications and features input as training data.

[0051] Next, the results output from the output layer 60c are input to the input layer 61a of the second neural network 61 constituting the trained second deep learning algorithm 61. Based on the input feature amount, the second deep learning algorithm 61 outputs the probability that the analysis target cell included in the analysis image belongs to each of the morphological cell classifications and characteristics input as training data from the output layer 61b. Furthermore, it is determined that the analysis target cell included in the analysis image belongs to the morphological classification with the highest value among these probabilities, and a label value linked to the morphological cell type or cell characteristic is output. The label value itself, or data in which the label value is replaced with information (e.g., terminology, etc.) indicating the presence or absence of the morphological cell type or cell characteristic, is output as data 83 related to the cell morphology. In FIG. 4, the label value "1" is output by the classifier from the analysis data 81 as the most likely label value 82, and the character data "segmented neutrophil" corresponding to this label value is output as data 83 related to the cell morphology. Reference numerals 60c and 61c in FIG. 4 denote intermediate layers.

[0052] [2. Image analysis system 1] <Image analysis system 1 configuration> A second embodiment of the present disclosure relates to an image analysis system. Referring to FIG. 5, the image analysis system according to the second embodiment includes a deep learning device 100A and an image analysis device 200A. The vendor-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A causes the neural network 50 to learn using training data, and provides the deep learning algorithm 60 trained by the training data to the user. The deep learning algorithm configured from the trained neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via a recording medium 98 or a network 99. The image analysis device 200A analyzes the image to be analyzed using the deep learning algorithm configured from the trained neural network 60.

[0053] The deep learning device 100A is, for example, a general-purpose computer, and performs deep learning processing based on a flowchart described later. The image analysis device 200A is, for example, a general-purpose computer, and performs image analysis processing based on a flowchart described later. The recording medium 98 is, for example, a computer-readable, non-transitory tangible recording medium, such as a DVD-ROM or a USB memory.

[0054] The deep learning device 100A is connected to an imaging device 300. The imaging device 300 includes an image sensor 301 and a fluorescent microscope 302, and captures a bright-field image of a training specimen 308 set on a stage 309. The training specimen 308 is stained as described above. The deep learning device 100A acquires a training image 70 captured by the imaging device 300.

[0055] The image analysis device 200A is connected to the imaging device 400. The imaging device 400 includes an image sensor 401 and a fluorescence microscope 402, and captures a bright-field image of a specimen 408 to be analyzed that is set on a stage 409. The specimen 408 to be analyzed has been stained in advance as described above. The image analysis device 200A acquires an image 78 to be analyzed that has been captured by the imaging device 400. The imaging devices 300 and 400 may be a known optical microscope or a virtual slide scanner having a function of capturing an image of a sample. <Hardware configuration of deep learning device>

[0056] Referring to FIG. 6, the vendor-side device 100 (deep learning device 100A, deep learning device 100B) includes a processing unit 10 (10A, 10B), an input unit 16, and an output unit 17.

[0057] The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing described later, a memory 12 used as a working area for data processing, a recording unit 13 that records a program and processing data described later, a bus 14 that transmits data between each unit, an interface unit 15 that inputs and outputs data to and from an external device, and a GPU (Graphics Processing Unit) 19. An input unit 16 and an output unit 17 are connected to the processing unit 10. Illustratively, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 11. That is, in the following description, the processing performed by the CPU 11 includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

[0058] In order to perform the processing of each step described below in Fig. 8, the processing unit 10 pre-records the program according to the present invention and the pre-trained neural network 50 in, for example, an executable format in the recording unit 13. The executable format is, for example, a format generated by conversion from a programming language by a compiler. The processing unit 10 uses the program recorded in the recording unit 13 to perform training processing of the pre-trained first neural network 50 and second neural network 51.

[0059] In the following description, unless otherwise specified, the processing performed by the processing unit 10 means the processing performed by the CPU 11 based on the programs and neural network 50 stored in the recording unit 13 or memory 12. The CPU 11 uses the memory 12 as a working area to temporarily store necessary data (intermediate data during processing, etc.), and records data to be stored for a long time, such as calculation results, in the recording unit 13 as appropriate.

[0060] <Hardware configuration of image analysis device> 7, the user side device 200 (image analysis device 200A, image analysis device 200B, image analysis device 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and an output unit 27.

[0061] The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing described later, a memory 22 used as a working area for data processing, a recording unit 23 that records a program and processing data described later, a bus 24 that transmits data between each unit, an interface unit 25 that inputs and outputs data to and from an external device, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. Illustratively, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 21. That is, in the following description, the processing performed by the CPU 21 includes the processing performed by the CPU 21 using the GPU 29 as an accelerator.

[0062] In addition, in order to perform the processing of each step described in the image analysis processing below, the processing unit 20 pre-records the program according to the present invention and a deep learning algorithm 60 of a trained neural network structure in, for example, an executable format in the recording unit 23. The executable format is, for example, a format generated by conversion from a programming language by a compiler. The processing unit 20 performs processing using the program recorded in the recording unit 23 and the first deep learning algorithm 60 and second deep learning algorithm 61.

[0063] In the following description, unless otherwise specified, the processing performed by the processing unit 20 means processing actually performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the recording unit 23 or memory 22. The CPU 21 uses the memory 22 as a working area to temporarily store necessary data (intermediate data during processing, etc.), and records data to be stored for a long time, such as calculation results, in the recording unit 23 as appropriate.

[0064] <Function blocks and processing procedures> (Deep learning processing) 8, the processing unit 10A of the deep learning device 100A according to this embodiment includes a training data generating unit 101, a training data input unit 102, and an algorithm updating unit 103. These functional blocks are realized by installing a program for causing a computer to execute deep learning processing in the recording unit 13 or memory 12 of the processing unit 10A and executing this program by the CPU 11. A training data database (DB) 104 and an algorithm database (DB) 105 are recorded in the recording unit 13 or memory 12 of the processing unit 10A.

[0065] The training images 70 are captured in advance by the imaging device 300 and stored in advance in the recording unit 13 or the memory 12 of the processing unit 10A. The first deep learning algorithm 50 and the second deep learning algorithm 51 are stored in advance in the algorithm database 105 in association with, for example, the morphological cell type or cell characteristics to which the cell to be analyzed belongs.

[0066] The processing unit 10A of the deep learning device 100A performs the processes shown in Fig. 9. Explaining this using the functional blocks shown in Fig. 8, the processes of steps S11, S12, S16, and S17 are performed by the training data generation unit 101. The process of step S13 is performed by the training data input unit 102. The processes of steps S14 and S18 are performed by the algorithm update unit 103. An example of the deep learning process performed by the processing unit 10A will be described with reference to FIG.

[0067] First, the processing unit 10A acquires the training image 70. The training image 70 is acquired by the operator by importing it from the imaging device 300, importing it from the recording medium 98, or via the network through the I / F unit 15. When acquiring the training image 70, information on which of the morphologically classified cell types and / or morphological cell characteristics the training image 70 represents is also acquired. The information on which of the morphologically classified cell types and / or morphological cell characteristics the training image 70 represents may be linked to the training image 70, or may be input by the operator through the input unit 16.

[0068] In step S11, the processing unit 10A converts the acquired training image 70 into luminance Y, first hue Cb, and second hue Cr, and generates color tone vector data 74 according to the procedure described in the training data generation method above.

[0069] In step S12, the processing unit 10A assigns a label value corresponding to the color tone vector data 74 based on information indicating a morphologically classified cell type and / or a cellular feature in the morphological classification, which is linked to the training image 70, and a label value linked to a morphologically classified cell type or a cellular feature in the morphological classification, which is stored in the memory 12 or the recording unit 13. In this way, the processing unit 10A generates training data 75.

[0070] 8, processing unit 10A trains first neural network 50 and second neural network 51 using training data 75. Training results of first neural network 50 and second neural network 51 are accumulated every time training is performed using a plurality of training data 75.

[0071] In the image analysis method according to the present embodiment, a convolutional neural network is used, and a stochastic gradient descent method is used, so in step S14, the processing unit 10A judges whether or not training results for a predetermined number of trials have been accumulated. If training results for the predetermined number of trials have been accumulated (YES), the processing unit 10A proceeds to processing in step S15, and if training results for the predetermined number of trials have not been accumulated (NO), the processing unit 10A proceeds to processing in step S16.

[0072] Next, in a case where the training results have been accumulated for a predetermined number of trials, in step S15, the processing unit 10A uses the training results accumulated in step S13 to update the connection weights w of the first neural network 50 and the second neural network 51. Since the image analysis method according to this embodiment uses the stochastic gradient descent method, the connection weights w of the first neural network 50 and the second neural network 51 are updated at a stage where the learning results for a predetermined number of trials have been accumulated. The process of updating the connection weights w is specifically a process of performing calculations by the gradient descent method shown in (Equation 11) and (Equation 12) described later.

[0073] In step S16, the processing unit 10A determines whether the first neural network 50 and the second neural network 51 have been trained with a prescribed number of training data 75. If training has been performed with a prescribed number of training data 75 (YES), the deep learning process is terminated.

[0074] If the first neural network 50 and the second neural network 51 have not been trained with the specified number of training data 75 (NO), the processing unit 10A proceeds from step S16 to step S17, and performs the processes from step S11 to step S16 on the next training image 70.

[0075] According to the above-described process, the first neural network 50 and the second neural network 51 are trained to obtain the second deep learning algorithm 60 and the second deep learning algorithm 62. (Neural network structure)

[0076] As described above, in this embodiment, a convolutional neural network is used. FIG. 10(a) illustrates the structures of the first neural network 50 and the second neural network 51. The first neural network 50 and the second neural network 51 include input layers 50a and 51a, output layers 50b and 51b, and intermediate layers 50c and 51c between the input layers 50a and 51a and the output layers 50b and 51b, and the intermediate layers 50c and 51c are composed of a plurality of layers. The number of layers constituting the intermediate layers 50c and 51c can be, for example, five or more layers.

[0077] In the first neural network 50 and the second neural network 51, a plurality of nodes 89 arranged in layers are connected between the layers, so that information is propagated in only one direction, as indicated by the arrow D in the figure, from the layers 50a and 51a on the input side to the layers 50b and 51b on the output side.

[0078] (Operation at each node) Fig. 10(b) is a schematic diagram showing the calculations at each node. Each node 89 receives multiple inputs and calculates one output (z). In the example shown in Fig. 10(b), node 89 receives four inputs. The total input (u) received by node 89 is expressed by the following (Equation 1).

[0079]

number

[0080] Each input is multiplied by a different weight. In (Equation 1), b is a value called the bias. The output (z) of the node is the output of a given function f for the total input (u) expressed in (Equation 1), and is expressed by the following (Equation 2). The function f is called the activation function.

[0081]

number

[0082] FIG. 10(c) is a schematic diagram showing the operation between nodes. In the neural network 50, nodes that output a result (z) expressed by (Equation 2) for a total input (u) expressed by (Equation 1) are arranged in layers. The output of a node in the previous layer becomes the input of a node in the next layer. In the example shown in FIG. 10(c), the output of a node 89a in the left layer becomes the input of a node 89b in the right layer. Each node 89b in the right layer receives the output from the node 89a in the left layer. A different weight is applied to each connection between each node 89a in the left layer and each node 89b in the right layer. If the outputs of each of the multiple nodes 89a in the left layer are x1 to x4, the inputs to each of the three nodes 89b in the right layer are expressed by the following (Equations 3-1) to (Equations 3-3).

[0083]

number

[0084]

number

[0085]

number

[0086] (Activation function) In the image analysis method according to the embodiment, a rectified linear unit function is used as the activation function. The rectified linear unit function is expressed by the following (Equation 5).

[0087]

number

[0088]

number

[0089] (Neural network training) If the function represented using a neural network is denoted as y(x:w), the function y(x:w) will change when the parameter w of the neural network is changed. Adjusting the function y(x:w) so that the neural network selects the parameter w that is more suitable for the input x is called neural network learning. Suppose multiple pairs of input and output of the function represented using a neural network are given. If the desired output for a certain input x is d, the input / output pair is {(x1,d1),(x2,d2),...,(x n ,d n )}. The set of pairs represented by (x,d) is called training data. Specifically, the set of pairs of color density values ​​for each pixel in the R, G, and B single-color images and labels of true-value images shown in Fig. 2(b) is the training data shown in Fig. 2(a). The learning of a neural network is based on what kind of input / output pairs (x n ,d n ), input x nGiven the neural network output y(x n :w) but the output is d n This means adjusting the weights w so that the weights are as close as possible to the error function. The error function is the closeness of the function expressed by the neural network to the training data.

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094] The target output d by the softmax function of (Eq. 7) n Let d be 1 only if the output is the correct class, and 0 otherwise. Let d be the target output. n =[d n1 , ,d nK ], for example, input x n If the correct class of is C3, the target output d n3 Only the input is 1, and the other target outputs are 0. When encoded in this way, the posterior distribution is expressed by the following (Equation 9).

[0095]

number

[0096]

number

[0097] Minimizing the error function E(w) with respect to the parameter w is the same as finding a local minimum of the function E(w). The parameter w is the weight of the connection between nodes. The minimum of the weight w is found by iterative calculations that start with an arbitrary initial value and repeatedly update the parameter w. One example of such calculations is the gradient descent method.

[0098] In the gradient descent method, a vector expressed by the following (Equation 11) is used.

number

[0099]

number

number

[0100] The calculation by (Equation 12) may be performed on all the training data (n=1, , N) or only on a part of the training data. A gradient descent method performed on only a part of the training data is called a stochastic gradient descent method. In the image analysis method according to the embodiment, the stochastic gradient descent method is used.

[0101] (Image analysis processing) 11 shows a functional block diagram of an image analyzing device 200A that performs image analysis processing from an analysis target image 78 to generating data 83 related to cell morphology. The processing unit 20A of the image analyzing device 200A includes an analysis data generating unit 201, an analysis data input unit 202, an analysis unit 203, and a cell nucleus region detecting unit 204. These functional blocks are realized by installing a program for causing a computer according to the present invention to execute image analysis processing in the recording unit 23 or memory 22 of the processing unit 20A and executing this program by the CPU 21. The training data database (DB) 104 and the algorithm database (DB) 105 are provided from the deep learning device 100A via a recording medium 98 or a network 99, and are recorded in the recording unit 23 or memory 22 of the processing unit 20A.

[0102] The image 78 to be analyzed is captured by the imaging device 400 and stored in the recording unit 23 or memory 22 of the processing unit 20A. The first deep learning algorithm 60 and the second deep learning algorithm 61 including the trained connection weights w are stored in the algorithm database 105 in association with the type of cell or the characteristics of the cell based on the morphological classification to which the cell to be analyzed belongs, and function as a program module that is a part of a program that causes a computer to execute image analysis processing. That is, the first deep learning algorithm 60 and the second deep learning algorithm 61 are used in a computer having a CPU and a memory, and are used to identify which of a plurality of cell types that belong to a predetermined cell group and are morphologically classified corresponds to the cell to be analyzed, and to generate data 83 regarding the morphology of the cell. The generated data is output as necessary. The CPU 21 of the processing unit 20A causes the computer to function to execute calculations or processing of specific information according to the purpose of use. Specifically, the CPU 21 of the processing unit 20A generates data 83 related to the morphology of the cell using the first deep learning algorithm 60 and the second deep learning algorithm 61 recorded in the recording unit 23 or the memory 22. The CPU 21 of the processing unit 20A inputs the analysis data 81 to the input layer 60a, and outputs the feature amount of the analysis image 78 calculated by the first deep learning algorithm 60 from the output layer 60b. The CPU 21 of the processing unit 20A inputs the feature amount output from the first deep learning algorithm 60 to the input layer 61a of the second deep learning algorithm, and outputs the label value of the cell type or cell feature based on the morphological classification to which the cell included in the analysis target image belongs from the output layer 61b. 11, the processes of steps S21 and S22 are performed by the analysis data generation unit 201. The processes of steps S23, S24, S25, and S27 are performed by the analysis data input unit 202. The process of step S26 is performed by the analysis unit 203.

[0103] An example of image analysis processing performed by the processing unit 20A from the analysis target image 78 to generating data 83 relating to the morphology of the cells will be described with reference to FIG.

[0104] First, the processing unit 20A acquires the analysis image 78. The analysis image 78 is acquired by a user's operation, by being imported from the imaging device 400, by being imported from the recording medium 98, or by being acquired via the I / F unit 25 over a network.

[0105] In step S21, similar to step S11 shown in FIG. 9, the acquired analysis image 78 is converted into luminance Y, first hue Cb, and second hue Cr, and color tone vector data 80 is generated according to the procedure described in the above-mentioned method for generating analysis data.

[0106] Next, in step S22, the processing unit 20A generates analysis data 81 from the color tone vector data 80 according to the procedure explained in the above-mentioned method for generating analysis data.

[0107] Next, in step S23, the processing unit 20A acquires the first deep learning algorithm and the second deep learning algorithm stored in the algorithm database 105.

[0108] Next, in step S24, the processing unit 20A inputs the analysis data 81 to the first deep learning algorithm. The processing unit 20A inputs the feature amount output from the first deep learning algorithm to the second deep learning algorithm according to the procedure described in the image analysis method above, and the second deep learning algorithm outputs a label value to which it is determined that the cell to be analyzed contained in the analysis image belongs. The processing unit 20A stores this label value in the memory 22 or the recording unit 23.

[0109] In step S25, processing unit 20A judges whether all of the initially acquired analysis images 78 have been classified. If classification of all analysis images 78 has been completed (YES), processing proceeds to step S26, where analysis results including data 83 relating to cell morphology are output. If classification of all analysis images 78 has not been completed (NO), processing proceeds to step S27, where processing from step S21 to step S25 is performed on analysis images 78 that have not yet been classified.

[0110] According to this embodiment, it is possible to identify the cell type and cell characteristics based on morphological classification regardless of the skill of the examiner, and variation in morphological examination can be suppressed.

[0111] <Computer Program> The present disclosure includes a computer program for performing image analysis to analyze cell morphology, which causes a computer to execute the processes of steps S11 to S17 and / or S21 to S27.

[0112] Furthermore, an embodiment of the present disclosure relates to a program product, such as a storage medium, that stores the computer program. That is, the computer program is stored in a storage medium, such as a hard disk, a semiconductor memory element such as a flash memory, an optical disk, etc. The storage format of the program in the storage medium is not limited as long as the presentation device can read the program. It is preferable that the storage in the storage medium is non-volatile.

[0113] [3. Image analysis system 2] <Image analysis system 2 configuration> Another aspect of the image analysis system will now be described. FIG. 13 shows a configuration example of the second image analysis system. The second image analysis system includes a user-side device 200, which operates as an integrated image analysis device 200B. The image analysis device 200B is configured, for example, by a general-purpose computer, and performs both the deep learning process and the image analysis process described in the image analysis system 1. In other words, the second image analysis system is a stand-alone system that performs deep learning and image analysis on the user side. In the second image analysis system, the integrated image analysis device 200B installed on the user side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment.

[0114] 13, image analysis device 200B is connected to imaging device 400. Imaging device 400 captures training images 70 during deep learning processing, and captures analysis target images 78 during image analysis processing.

[0115] <Hardware configuration> The hardware configuration of the image analyzing device 200B is similar to the hardware configuration of the user side device 200 shown in FIG.

[0116] <Function blocks and processing procedures> FIG. 14 shows a functional block diagram of the image analysis device 200B. The processing unit 20B of the image analysis device 200B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a cell nucleus detection unit 204. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing and image analysis processing in the recording unit 23 or memory 22 of the processing unit 20B and executing this program by the CPU 21. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 23 or memory 22 of the processing unit 20B, and both are used in common during deep learning and image analysis processing. The first neural network 60 and the second neural network 61 that have been trained are associated with the type of cell or the characteristics of the cell based on, for example, the morphological classification to which the cells to be analyzed belong, and are stored in advance in the algorithm database 105. The connection weight w is updated by the deep learning process, and is stored in the algorithm database 105 as the deep learning algorithm 60. The training image 70 is assumed to be captured in advance by the imaging device 400, and to be stored in advance in the training data database (DB) 104, or the recording unit 23 or memory 22 of the processing unit 20B. The analysis target image 78 of the specimen to be analyzed is assumed to be captured in advance by the imaging device 400, and to be stored in advance in the recording unit 23 or memory 22 of the processing unit 20B.

[0117] The processing unit 20B of the image analysis device 200B performs the process shown in Fig. 9 during deep learning processing, and performs the process shown in Fig. 12 during image analysis processing. Explaining using the functional blocks shown in Fig. 13, during deep learning processing, the processes of steps S11, S12, S16, and S17 are performed by the training data generation unit 101. The process of step S13 is performed by the training data input unit 102. The processes of steps S14 and S18 are performed by the algorithm update unit 103. During image analysis processing, the processes of steps S21 and S22 are performed by the analysis data generation unit 201. The processes of steps S23, S24, S25, and S27 are performed by the analysis data input unit 202. The process of step S26 is performed by the analysis unit 203.

[0118] The procedure of the deep learning process and the procedure of the image analysis process performed by the image analysis device 200B are similar to the procedures performed by the deep learning device 100A and the image analysis device 200A, respectively. However, the image analysis device 200B acquires the training images 70 from the imaging device 400.

[0119] In the image analysis device 200B, the user can check the classification accuracy of the classifier. In the unlikely event that the classification result of the classifier differs from the classification result based on the user's image observation, the first deep learning algorithm and the second deep learning algorithm can be trained and corrected using the analysis data 81 as training data 78 and the classification result based on the user's image observation as the label value 77. In this way, the training efficiency of the first neural network 50 and the first neural network 51 can be further improved.

[0120] [3. Image analysis system 3] <Image analysis system 3 configuration> Another aspect of the image analysis system will now be described. FIG. 15 shows a configuration example of the third image analysis system. The third image analysis system includes a vendor-side device 100 and a user-side device 200. The vendor-side device 100 operates as an integrated image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is, for example, a general-purpose computer, and is a cloud server-side device that performs both the deep learning process and the image analysis process described in the image analysis system 1. The terminal device 200C is, for example, a general-purpose computer, and is a user-side terminal device that transmits an image to be analyzed to the image analysis device 100B through a network 99 and receives an image of the analysis result from the image analysis device 100B through the network 99.

[0121] In the third image analysis system, an integrated image analysis device 100B installed on the vendor side serves as both the deep learning device 100A and the image analysis device 200A. On the other hand, the third image analysis system includes a terminal device 200C, and provides an input interface for an analysis image 78 and an output interface for an analysis result image to the user-side terminal device 200C. In other words, the third image analysis system is a cloud service-type system in which the vendor side performing deep learning processing and image analysis processing provides an input interface for providing an analysis image 78 to the user side and an output interface for providing data 83 related to cell morphology to the user side. The input interface and the output interface may be integrated.

[0122] The image analysis device 100B is connected to the imaging device 300, and acquires training images 70 captured by the imaging device 300.

[0123] The terminal device 200C is connected to the imaging device 400, and acquires an analysis target image 78 captured by the imaging device 400.

[0124] <Hardware configuration> The hardware configuration of the image analyzing device 100B is similar to that of the vendor-side device 100 shown in Fig. 6. The hardware configuration of the terminal device 200C is similar to that of the user-side device 200 shown in Fig. 7. <Function blocks and processing procedures> FIG. 16 shows a functional block diagram of the image analysis device 100B. The processing unit 10B of the image analysis device 100B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a cell nucleus region detection unit 204. These functional blocks are realized by installing a program for causing a computer to execute deep learning processing and image analysis processing in the recording unit 13 or memory 12 of the processing unit 10B and executing this program by the CPU 11. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 13 or memory 12 of the processing unit 10B, and both are used in common during deep learning and image analysis processing. The first neural network 50 and the second neural network 51 are pre-stored in the algorithm database 105, for example, in association with the cell type or cell characteristics based on the morphological classification to which the cell being analyzed belongs, and the connection weight w is updated by deep learning processing and stored in the algorithm database 105 as the first deep learning algorithm 60 and the second deep learning algorithm 61.

[0125] The training images 70 are captured in advance by the imaging device 300, and are stored in advance in the training data database (DB) 104, or the recording unit 13 or memory 12 of the processing unit 10B. The analysis target image 78 is captured by the imaging device 400, and is stored in advance in the recording unit 23 or memory 22 of the processing unit 20C of the terminal device 200C.

[0126] The processing unit 10B of the image analysis device 100B performs the process shown in FIG. 9 during deep learning processing, and performs the process shown in FIG. 12 during image analysis processing. Explaining using the functional blocks shown in FIG. 16, during deep learning processing, the processes of steps S11, S12, S16, and S17 are performed by the training data generation unit 101. The process of step S13 is performed by the training data input unit 102. The processes of steps S14 and S18 are performed by the algorithm update unit 103. During image analysis processing, the processes of steps S21 and S22 are performed by the analysis data generation unit 201. The processes of steps S23, S24, S25, and S27 are performed by the analysis data input unit 202. The process of step S26 is performed by the analysis unit 203.

[0127] The deep learning processing procedure and the image analysis processing procedure performed by the image analysis device 100B are similar to the procedures performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively.

[0128] The processing unit 10B receives the analysis target image 78 from the user terminal device 200C, and generates the training data 75 according to steps S11 to S17 shown in FIG.

[0129] 12, the processing unit 10B transmits the analysis result including the data 83 on the cell morphology to the user-side terminal device 200C. In the user-side terminal device 200C, the processing unit 20C outputs the received analysis result to the output unit 27.

[0130] As described above, the user of terminal device 200C can obtain data 83 relating to cell morphology as the analysis result by transmitting analysis target image 78 to image analyzing device 100B.

[0131] According to the image analyzing device 100B according to the third embodiment, a user can use a classifier without acquiring the training data database 104 and the algorithm database 105 from the deep learning device 100A. This makes it possible to provide a service for identifying cell types and cell features based on morphological classification as a cloud service.

[0132] [4. Other forms] Although the present invention has been described above with reference to the outline and specific embodiments, the present invention is not limited to the outline and each embodiment described above.

[0133] In the present disclosure, a method of generating training data 75 by converting color tones into luminance Y, a first hue Cb, and a second hue Cr is illustrated, but the conversion of color tones is not limited to this. For example, the three primary colors of red (R), green (G), and blue (B) may be used without converting the color tones. Alternatively, two primary colors may be used by removing one hue from the primary colors. Alternatively, one primary color may be only one of the three primary colors of red (R), green (G), and blue (B) (for example, green (G)). The primary colors may be converted into three primary colors of cyan (C), magenta (M), and yellow (Y). For example, the analysis target image 78 is not limited to a color image of the three primary colors of red (R), green (G), and blue (B), but may be a color image of two primary colors, or may be an image including one or more primary colors.

[0134] In the above-mentioned training data generating method and analysis data generating method, in step S11, the processing units 10A, 20B, 10B generate the color tone matrices 72y, 72cb, 72cr from the training image 70, but the training image 70 may be an image method converted into luminance Y, a first hue Cb, and a second hue Cr. That is, the processing units 10A, 20B, 10B may directly acquire the luminance Y, the first hue Cb, and the second hue Cr from, for example, a virtual slide scanner from the beginning. Similarly, in step S21, the processing units 20A, 20B, 10B generate the color tone matrices 72y, 72cb, 72cr from the analysis target image 78, but the processing units 20A, 20B, 10B may directly acquire the luminance Y, the first hue Cb, and the second hue Cr from, for example, a virtual slide scanner from the beginning.

[0135] In addition to RGB and CMY, YUV and CIE L * a * b * Various color spaces, such as RGB, ...

[0136] In the color tone vector data 74 and the color tone vector data 80, the color tone information is stored for each pixel in the order of brightness Y, first hue Cb, and second hue Cr, but the order in which the color tone information is stored and handled is not limited to this. However, it is preferable that the order of the color tone information in the color tone vector data 74 and the order of the color tone information in the color tone vector data 80 are the same.

[0137] In each field image analysis system, the processing units 10A and 10B are realized as an integrated device, but the processing units 10A and 10B do not have to be an integrated device, and the CPU 11, memory 12, recording unit 13, GPU 19, etc. may be located in different places and connected by a network. The processing units 10A and 10B, the input unit 16, and the output unit 17 do not necessarily have to be located in one place, and may be located in different places and connected to each other so as to be able to communicate with each other by a network. The processing units 20A, 20B, and 20C are similar to the processing units 10A and 10B.

[0138] In the above first to third embodiments, each of the functional blocks of the training data generation unit 101, the training data input unit 102, the algorithm update unit 103, the analysis data generation unit 201, the analysis data input unit 202, and the analysis unit 203 is executed by a single CPU 11 or a single CPU 21, but each of these functional blocks does not necessarily have to be executed by a single CPU, and may be executed in a distributed manner by multiple CPUs. Also, each of these functional blocks may be executed in a distributed manner by multiple GPUs, or may be executed in a distributed manner by multiple CPUs and multiple GPUs.

[0139] In the second and third embodiments, a program for performing the processing of each step described in Fig. 9 and Fig. 12 is pre-recorded in the recording units 13 and 23. Alternatively, the program may be installed in the processing units 10B and 20B from a computer-readable, non-transitory, tangible recording medium 98, such as a DVD-ROM or a USB memory. Alternatively, the processing units 10B and 20B may be connected to a network 99, and the program may be downloaded from, for example, an external server (not shown) via the network 99 and installed.

[0140] In each image analysis system, the input units 16, 26 are input devices such as a keyboard or a mouse, and the output units 17, 27 are realized as display devices such as a liquid crystal display. Alternatively, the input units 16, 26 and the output units 17, 27 may be integrated into a touch panel display device. Alternatively, the output units 17, 27 may be configured as a printer or the like.

[0141] In each of the image analysis systems described above, imaging device 300 is directly connected to deep learning device 100A or image analysis device 100B, but imaging device 300 may be connected to deep learning device 100A or image analysis device 100B via network 99. Similarly, imaging device 400 is directly connected to image analysis device 200A or image analysis device 200B, but imaging device 400 may be connected to image analysis device 200A or image analysis device 200B via network 99.

[0142] [5. Effects of deep learning algorithms] In order to verify the effect of the deep learning algorithm, a comparison was made between the accuracy of cell identification between a conventional cell identification method using machine learning and the cell identification method using the deep learning algorithm of the present disclosure. Peripheral blood smears were prepared using the SP-1000i smear preparation device, and the cells were imaged using the DI-60 automated blood image analyzer. The staining was May-Giemsa staining. The conventional machine learning method for identifying cells was performed using an automated blood image analyzer, DI-60. Validation was performed by having three people, including a doctor and an experienced laboratory technician, observe the images. The results of a comparison of blood cell classification accuracy are shown in Figure 17. When the deep learning algorithm was used, discrimination was achieved with higher accuracy than the conventional method.

[0143] Next, we investigated whether the deep learning algorithm of the present disclosure could identify morphological features observed in myelodysplastic syndrome (MDS). The results are shown in Figure 18.

[0144] As shown in FIG. 18, morphological nuclear abnormalities, vacuolization, granule distribution abnormalities, etc. could be accurately identified.

[0145] From the above results, it is believed that the deep learning algorithm disclosed herein can accurately identify cell types based on morphological classification and cell characteristics. [Explanation of symbols]

[0146] 200 Image analysis device 10 Processing section 50,51 Pre-training deep learning algorithms 50a, 51a Input layer 50b,51b output layer 60,61 Pre-trained deep learning algorithms 75 training data 80 Analysis Data 83 Cell morphology data

Claims

1. A computer-implemented image analysis method for analyzing cell morphology using a deep learning algorithm having a neural network structure, comprising: generating analysis data including color tone vector data combining the brightness of each pixel and the gradation values ​​of at least two kinds of hues of each pixel from the data of the image of the stained blood cells; inputting the analysis data, which includes information about the cells to be analyzed, into the deep learning algorithm; Analyzing the analysis data using the deep learning algorithm to calculate the probability that the cell to be analyzed belongs to each of a plurality of morphological classifications of cells belonging to a predetermined cell group; The predetermined cell population is a blood cell population. Image analysis methods.

2. The image analysis method according to claim 1 , further comprising identifying the cell to be analyzed as belonging to one of a plurality of morphological classifications of cells belonging to a predetermined cell group based on the calculated probability.

3. The image analysis method according to claim 1 , wherein the predetermined cell group is a group of cells belonging to a predetermined cell lineage.

4. The image analysis method according to claim 3 , wherein the predetermined cell lineage is a hematopoietic cell lineage.

5. The image analysis method according to claim 1 , wherein the morphological classification is identification of a type of the cell to be analyzed.

6. 6. The image analysis method according to claim 1, wherein the morphological classification of the plurality of cells includes a plurality of cells selected from the group consisting of segmented neutrophils, band neutrophils, metamyelocytes, myelocytes, blasts, lymphocytes, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts, giant platelets, platelet aggregates, and megakaryocytes.

7. The image analysis method according to claim 1 , wherein the morphological classification is identification of abnormal findings in the cells to be analyzed.

8. The image analysis method according to any one of claims 1 to 7, wherein the morphological classification of the plurality of cells includes at least one cell selected from the group consisting of cells with morphological nuclear abnormalities, presence of vacuoles, granule morphological abnormalities, abnormal granule distribution, presence of abnormal granules, abnormal cell size, presence of inclusion bodies, and bare nuclei.

9. The deep learning algorithm comprises: a first algorithm for calculating a probability that the target cell belongs to each of a first morphological classification of a plurality of cells belonging to a predetermined cell group; The image analysis method according to claim 1 , further comprising: a second algorithm for calculating a probability that the cell being analyzed belongs to each of second morphological classifications of a plurality of cells belonging to a predetermined cell group.

10. The first morphological classification is a type of the cell to be analyzed, The image analysis method according to claim 9 , wherein the second morphological classification is an abnormal finding of the cells to be analyzed.

11. The image analysis method according to claim 1 , wherein the staining is selected from Wright's staining, Giemsa staining, Wright-Giemsa staining, and May-Giemsa staining.

12. An image analysis device that analyzes cell morphology using a deep learning algorithm having a neural network structure, comprising: a processor that generates analysis data including color vector data combining the brightness of each pixel and the gradation values ​​of at least two hues of each pixel from image data of stained blood cells, inputs the analysis data including information about a target cell to the deep learning algorithm, analyzes the analysis data using the deep learning algorithm, and calculates a probability that the target cell belongs to each of the morphological classifications of a plurality of cells belonging to the predetermined cell group; The predetermined cell population is a blood cell population. Image analysis device.

13. The image analysis device according to claim 12 , wherein the morphological classification is identification of a type of the cell to be analyzed.

14. The image analyzing device according to claim 12 or 13, wherein the morphological classification is identification of abnormal findings in the cells to be analyzed.

15. A computer program for image analysis that analyzes cell morphology using a deep learning algorithm having a neural network structure, comprising: generating analytical data including color vector data combining the brightness of each pixel and the gradation values ​​of at least two hues of each pixel from data of an image of stained blood cells; inputting the analytical data including information about the cells to be analyzed into a classifier including the deep learning algorithm; analyzing the analytical data using the deep learning algorithm; and calculating the probability that the cells to be analyzed belong to each of the morphological classifications of a plurality of cells belonging to the predetermined cell group; The predetermined cell population is a blood cell population. Have a computer carry out the process, Computer program.

16. The computer program product of claim 15 , wherein the morphological classification is identification of the type of the analyzed cell.

17. The computer program according to claim 15 or 16, wherein the morphological classification is the identification of abnormal findings in the cells to be analyzed.

Images