Image analysis method, device, storage medium and deep learning algorithm generation method

Abstract

An object of the present invention is to provide an image analysis method for more accurately identifying the morphology of each of a plurality of cells contained in an image for analysis. The image analysis method for analyzing the morphology of cells using a deep learning algorithm having a neural network structure inputs analysis data containing information related to the analysis target cell into the deep learning algorithm (60, 61), and calculates the probability of each of the morphological classifications of the plurality of cells belonging to the certain cell group by the deep learning algorithm (60, 61), and analyzes the image by this method.

Description

Technical Field

[0001] The present invention relates to an image analysis method, device, storage medium and a method for generating a deep learning algorithm for analyzing cell morphology. Background Art

[0002] Patent document 1 discloses a cell recognition system for processing microscope images. In the cell recognition system, a model trained using a machine training technique associates pixels in an acquired image with one or more of cells, cell boundaries, backgrounds, and their equivalents. The machine training technique uses a random forest decision tree technique.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent document 1: Japanese Patent Application No. 2016-534709. Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] In cell examination, the examiner usually observes the cells under a microscope and identifies the type and characteristics of the cells morphologically. However, cells of the same lineage have similar morphologies, so in order to identify cells morphologically, it is necessary to observe many cell specimens and improve identification skills. Skill is particularly required when identifying abnormal cells that appear when a disease occurs. For example, when the frequency of abnormal cells is low, such as in the early stage of myelodysplastic syndrome, there is a risk that the examiner with insufficient skills will not notice the abnormal cells.

[0008] In addition, the number of specimens that an inspector can observe in one day is limited, and observing more than 100 specimens in one day will also place a burden on the inspector.

[0009] Flow-type automatic blood cell sorting devices can also increase the number of cell inspections. However, the information obtained from flow-type automatic blood cell sorting devices is limited, and it is difficult to identify blood cells with low frequency of occurrence, such as blasts, promyelocytes, and giant platelets.

[0010] As shown in the method described in Patent Document 1, a method of identifying cells using machine training techniques (also called machine learning) is also known. However, users are required to create training data for training machine learning models, and a huge amount of labor is required to generate models. In addition, since users create training data, the amount of training data that can be created is limited, and there are still problems with the accuracy and generalizability of the analysis of machine learning models.

[0011] Furthermore, the method described in Patent Document 1 is a method for distinguishing between cellular parts and non-cellular parts in a microscope image, and therefore it is not possible to distinguish the type of each cell and what abnormalities it contains.

[0012] An object of the present invention is to provide an image analysis method for more accurately identifying the morphology of each of a plurality of cells included in an analysis image.

[0013] Technical means to solve technical problems

[0014] One embodiment of the present invention relates to an image analysis method. In the image analysis method, analysis data (80) is input into a deep learning algorithm (60, 61) having a neural network structure, wherein the analysis data is generated from an image of an analysis target cell and includes information related to the analysis target cell; the probability of each item in the morphological classification of several cells belonging to a certain cell group being the analysis target cell is calculated by the deep learning algorithm. According to this embodiment, even if the examiner does not perform microscopic observation, the probability of each item in the morphological classification of several cells belonging to a certain cell group being the analysis target cell can be obtained and the image can be analyzed.

[0015] A preferred technical solution is that the image analysis method identifies the morphological category to which the analysis target cell belongs based on the calculated probability. According to this embodiment, even if the examiner does not observe the cell under a microscope, he or she can identify which morphological category the analysis target cell matches.

[0016] A preferred technical solution is that: the certain cell group is a group of blood cells. According to this embodiment, even if the examiner does not observe under a microscope, the morphological classification of blood cells can be performed.

[0017] A preferred technical solution is that the certain cell group is a group of cells belonging to a certain cell line. A more preferred technical solution is that the certain cell line is a hematopoietic system. According to this embodiment, even if the examiner does not observe under a microscope, the morphological classification of cells belonging to the same cell line can be performed.

[0018] A preferred technical solution is that each morphological classification indicates the type of cell. A more preferred technical solution is that the morphological classification includes the following: neutrophils including segmented neutrophils and band-shaped neutrophils, metamyelocytes, myelocytes, promyelocytes, blasts, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts, giant platelets, platelet aggregation, and megakaryocytes. According to this embodiment, cells of the same series with similar morphology can also be identified.

[0019] A preferred technical solution is that each morphological classification indicates abnormal behavior of a cell, and a more preferred technical solution is that the morphological classification includes at least one cell selected from the group consisting of the following items: abnormal nuclear morphology, presence of vacuoles, abnormal granule morphology, abnormal granule distribution, presence of abnormal granules, abnormal cell size, presence of inclusion bodies, and naked nuclei. According to this embodiment, cells showing abnormal behavior can also be identified.

[0020] In the embodiment, the data related to the cell morphology are data related to the type of cells in morphological classification and data related to the characteristics of cells in morphological classification. Through this embodiment, the morphological types of cells and the characteristics of cells can be output.

[0021] The preferred technical solution is: in the embodiment, the deep learning algorithm includes: a first algorithm, which calculates the probability of each item in the first morphological classification of several cells belonging to a certain cell group; a second algorithm, which calculates the probability of each item in the second morphological classification of several cells belonging to a certain cell group. For example, the first morphological classification is the type of the cell to be analyzed, and the second morphological classification is the abnormal performance of the cell to be analyzed. In this way, the recognition accuracy of cells with similar morphology can be further improved.

[0022] In the embodiment, the analysis data (80) is generated from the image of the stained blood cells taken. A more preferred technical solution is that the staining is selected from Wright's staining, Giemsa staining, Swiss Giemsa staining and Mei Giemsa staining. In this way, the same identification as the conventional observation under the microscope can be performed.

[0023] The analysis data (80) and training data (75) include information related to brightness of the analysis target image and the training image and information related to at least two hues. This can improve recognition accuracy.

[0024] Other embodiments of the present invention relate to an image analysis device (200). The image analysis device (200) comprises: a processing unit (10), which inputs analysis data (80) into a deep learning algorithm (60, 61) having a neural network structure, wherein the analysis data is generated by an image of an analysis target cell and contains relevant information of the analysis target cell, and the probability of each item in the morphological classification of several cells under a certain cell group is calculated by the deep learning algorithm. A preferred technical solution is that each morphological classification is an identification of the type of the analysis target cell. Another preferred technical solution is that each morphological classification is an identification of an abnormal manifestation of the analysis target cell.

[0025] Another embodiment of the present invention relates to a storage medium, which is a non-transitory computer-readable storage medium storing a program executed by a processor. The program causes a computer (200) to perform the following processing: inputting analysis data (80) into a deep learning algorithm (60, 61) having a neural network structure, wherein the analysis data (80) is generated from an image of an analysis target cell and contains relevant information of the analysis target cell, and calculating the probability of each item of the morphological classification of the analysis target cell belonging to a plurality of cells under a certain cell group through the deep learning algorithm.

[0026] Other embodiments of the present invention relate to a method for generating a trained deep learning algorithm (60, 61). In this embodiment, training data containing cell-related information is input into an input layer (50a, 51a) of a neural network (50, 51), and a label value is input as an output layer (50b, 51b), and the label value is associated with each morphological classification of several cells under a certain cell population. The preferred technical solution is that each morphological classification indicates the type of cell or the abnormal performance of the cell.

[0027] By using an image analysis device (200) and a trained deep learning algorithm (60, 61), morphological cell types and cell characteristics can be identified without being affected by the skills of an examiner.

[0028] Effects of the Invention

[0029] The morphology of each of the multiple cells included in the analysis image can be identified. As a result, cell inspection can be performed regardless of the inspector's skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram showing an overview of the present invention;

[0031] Figure 2 A schematic diagram of an example of a step of generating training data, a first deep learning algorithm, and a training step of a second deep learning algorithm;

[0032] Figure 3 Examples of representing label values;

[0033] Figure 4 Schematic diagram of the steps for generating parsed data and an example of the steps for cell identification using a deep learning algorithm;

[0034] Figure 5 A schematic diagram showing a configuration example of the image analysis system 1;

[0035] Figure 6 A block diagram showing an example of the hardware configuration of the manufacturer-side device 100;

[0036] Figure 7A block diagram showing an example of the hardware configuration of the user-side device 200;

[0037] Figure 8 A block diagram for explaining an example of the functions of the deep learning device 100A;

[0038] Fig. 9 A flowchart showing an example of the flow of a deep learning process;

[0039] Fig.10 Schematic diagrams used to illustrate neural networks;

[0040] Fig.11 A block diagram for explaining an example of the functions of the image analysis device 200A;

[0041] Fig.12 A flowchart showing an example of the flow of image analysis processing;

[0042] Fig.13 A schematic diagram showing a configuration example of the image analysis system 2;

[0043] Fig.14 A block diagram for explaining an example of the functions of an integrated image analysis device 200B;

[0044] Fig.15 A schematic diagram showing a configuration example of the image analysis system 3;

[0045] Fig.16 A block diagram for explaining an example of the functions of an integrated image analysis device 100B;

[0046] Fig.17 Indicates the identification results of cell types obtained using deep learning algorithms;

[0047] Fig.18 Represents the recognition results of cell features obtained using deep learning algorithms. DETAILED DESCRIPTION

[0048] The following describes the outline and embodiments of the present invention in detail with reference to the accompanying drawings. In addition, the same reference numerals in the following description and the accompanying drawings represent the same or similar components, and thus the description of the same or similar components is omitted.

[0049] [1. Image analysis method]

[0050] The first embodiment of the present invention relates to an image analysis method for analyzing cell morphology. The image analysis method inputs analysis data including information related to the analysis target cell into a classifier including a deep learning algorithm having a neural network structure. The classifier calculates the probability that the analysis target cell belongs to each of the morphological classifications of several cells under the certain cell group. The image analysis method preferably further includes the following processing: based on the probability, identifying which of the morphological classifications of several cells under the certain cell group the analysis target cell belongs to.

[0051] In the first embodiment, the cells to be analyzed belong to a certain cell group. The certain cell group is a cell group that constitutes each organ in the body of a mammal or bird. Under normal circumstances, the certain cell group contains several cell types that are morphologically classified by histological microscopy and cytological microscopy. Morphological classification (also called "morphological classification") includes the classification of cell types and the classification of morphological characteristics of cells. Preferably, the cells to be analyzed are a cell group belonging to a certain cell line under the certain cell group. A certain cell line refers to a cell group belonging to the same line that is differentiated from a certain tissue stem cell. The certain cell line is preferably a hematopoietic system, and more preferably a cell in the blood (also called "blood cell").

[0052] The existing method is to classify the hematopoietic system cells morphologically by observing the specimens stained for bright field under the bright field of a microscope. Preferably, the stain is selected from Wright's stain, Giemsa stain, Swiss Giemsa stain and Mei Giemsa stain. More preferably, Mei Giemsa stain. The specimen is not particularly limited as long as it satisfies the requirement of being able to observe the morphology of each cell belonging to a certain cell group individually. For example, smear specimens and printed specimens can be listed. Preferably, a smear specimen using peripheral blood or bone marrow as a sample is used, and more preferably a smear specimen using peripheral blood.

[0053] In morphological classification, blood cells include the following cell types: neutrophils including segmented neutrophils and band-shaped neutrophils, metamyelocytes, granulocytes, promyelocytes, blasts, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts (nucleated erythrocytes, including proerythroblasts, basophilic erythroblasts, polychromatic erythroblasts, normochromatic erythroblasts, promegablasts, basophilic megablasts, polychromatic megablasts and normochromatic megablasts), giant platelets, platelet aggregation and megakaryocytes (nucleated megakaryocytes, including micromegakaryocytes), etc.

[0054] In addition, the certain cell population may also include abnormal cells showing morphological abnormalities in addition to normal cells. Abnormality is manifested as cell characteristics classified morphologically. Abnormal cells are, for example, cells that appear when suffering from certain diseases, such as tumor cells. In the hematopoietic system, certain diseases are selected from the group consisting of myelodysplastic syndrome, leukemia (including acute myeloid leukemia, acute myeloid leukemia (with neutrophil differentiation), acute promyelocytic leukemia, acute myelomonocytic leukemia, acute monocytic leukemia, erythroleukemia, acute primitive megakaryocyte leukemia, acute myeloid leukemia, acute lymphocytic leukemia, lymphoblastic leukemia, chronic myeloid leukemia and chronic lymphocytic leukemia, etc.), malignant lymphoma (Hodgkin's lymphoma and non-Hodgkin's lymphoma, etc.) and multiple myeloma. In addition, in the hematopoietic system, cells with abnormal manifestations are cells having at least one morphological characteristic selected from the group consisting of the following items: abnormal nuclear morphology, the presence of vacuoles, abnormal granule morphology, abnormal granule distribution, the presence of abnormal granules, abnormal cell size, the presence of inclusion bodies and naked nuclei.

[0055] Nuclear morphological abnormalities include: smaller nucleus; larger nucleus; excessive nuclear lobes; nuclei that should be lobed are not lobed normally (including pseudo-Pelger's nuclear abnormalities, etc.); vacuoles; nucleolar hypertrophy; nuclear gaps; one cell should have one nucleus but has two nuclear abnormalities, etc.

[0056] The abnormalities in the overall morphology of the cell can be listed as follows: vacuoles in the cytoplasm (also called vacuolar degeneration); abnormal morphology of granules such as azurophilic granules, neutrophilic granules, eosinophilic granules, and basophilic granules; abnormal distribution of the granules (excess, decrease, or disappearance); abnormal granules (such as toxic granules); abnormal cell size (larger or smaller than normal); inclusion bodies (Dühler bodies and rod-shaped bodies, etc.); naked nuclei, etc.

[0057] <Overview of image analysis method>

[0058] use Figure 1 An overview of the image analysis method is described.

[0059] The identifier used in the image analysis method includes several deep learning algorithms (sometimes just called "algorithms") having a neural network structure. The preferred technical solution is: including a first deep learning algorithm 50 and a second deep learning algorithm 51, the first deep learning algorithm 50 extracts the feature value of the cell, and the second deep learning algorithm 51 identifies the analysis target cell based on the feature value extracted by the first deep learning algorithm. The more preferred technical solution is: Figure 1As shown, the identifier contains not only the second deep learning algorithm downstream of the first deep learning algorithm, but also several deep learning algorithms trained according to the recognition purpose (sometimes numbered as the second, third, fourth, fifth...ith). For example, the second deep learning algorithm recognizes the types of cells based on the above-mentioned morphological classification. In addition, for example, the second deep learning algorithm recognizes the features of the above-mentioned cells based on morphological classification according to each feature. The preferred technical solution is: the first deep learning algorithm is a convolutionally connected neural network, and the second and subsequent deep learning algorithms located downstream of the first deep learning algorithm are fully connected neural networks.

[0060] Next use Figure 2~Figure 4 The example of the training data 75 is used to illustrate the generation method and the image analysis method. For the convenience of explanation, the first deep learning algorithm and the second deep learning algorithm are used for explanation below.

[0061] <Generation of training data>

[0062] The training image 70 used for training the deep learning algorithm is an image obtained by photographing cells whose types and characteristics of cells corresponding to the analysis target cells are known based on morphological classification. It is preferred that a sample containing cells of the same type as the analysis target cells be used and that the specimen used for photographing the training image 70 be prepared by the same specimen preparation method and staining method as the specimen containing the analysis target cells. In addition, it is preferred that the training image 70 be photographed under the same conditions as the conditions for photographing the analysis target cells.

[0063] For example, the training image 70 can be acquired in advance for each cell using an imaging device such as a well-known optical microscope or a virtual slide scanner. Figure 2 In the example of , the training image 70 is an original image taken by Sysmex DI-60 at 360 pixels × 365 pixels, which is reduced to 255 pixels × 255 pixels, but it is not necessary to reduce it. The number of pixels of the training image 70 is not particularly limited as long as it can be analyzed, and it is preferably more than 100 pixels on one side. In addition, Figure 2 In the example, there are red blood cells around the neutrophils, and the image can be cropped to retain only the target cells. An image can be used as a training image 70 if it contains at least one training target cell (which can contain red blood cells and normal-sized platelets) and the pixels belonging to the training target cell account for more than about 1 / 9 of the total pixels of the image.

[0064] As an example, the preferred technical solution is: In this embodiment, the shooting device is photographed using RGB colors and CMY colors. Preferably, the color image represents the depth or brightness of each primary color such as red, green and blue or cyan, magenta, and yellow with a 24-bit value (8 bits × 3 colors). The training image 70 only needs to include at least one hue and the depth or brightness of the hue, and more preferably includes at least two hues and the depth or brightness of each hue. Information including the hue and the depth or brightness of the hue is also called color tone.

[0065] Next, the color tone information of each pixel is converted from RGB color to a format that includes brightness information and hue information. The formats that include brightness information and hue information include YUV (YCbCr, YPbPr, YIQ, etc.). Here, the conversion to YCbCr format is taken as an example. Here, because the training image is RGB color, it is converted to brightness 72Y, the first hue (such as blue) 72Cb, and the second hue (such as red) 72Cr. RGB can be converted to YCbCr using well-known methods. For example, it can be converted from RGB to YCbCr in accordance with the international standard ITU-R BT.601. For example Figure 2 As shown, the converted brightness 72Y, the first hue 72Cb and the second hue 72Cr can be represented by a matrix of gradation values ​​(hereinafter also referred to as a hue matrix 72y, 72cb, 72cr). The brightness 72Y, the first hue 72Cb and the second hue 72Cr are represented by 256 gradations from 0 to 255. Here, the training image can also be converted using the three primary colors of red R, green G, and blue B and the three primary colors of cyan C, magenta M, and yellow Y instead of the brightness 72Y, the first hue 72Cb and the second hue 72Cr.

[0066] Next, based on the hue matrix 72y, 72cb, and 72cr, hue vector data 74 is generated for each pixel by combining three gradation values ​​of the brightness 72y, the first hue 72cb, and the second hue 72cr.

[0067] Next, for example, Figure 2 The training image 70 is a segmented neutrophil, so from Figure 2 Each tone vector data 74 generated from the training image 70 is assigned “1” as a label value 77 indicating a segmented neutrophil, thereby obtaining training data 75 . Figure 2 In FIG. 1 , the training data 75 is represented by 3 pixels×3 pixels for convenience, but in reality, there is hue vector data corresponding to the number of pixels when the training image 70 is captured.

[0068] Figure 3 7 shows an example of a label value 77. Regarding the label value, a different label value 77 is given depending on the type of cell and the presence or absence of a characteristic of each cell.

[0069] <Overview of Deep Learning>

[0070] by Figure 2 The outline of neural network training is explained as an example. The first neural network 50 and the second neural network 51 are preferably convolutional neural networks. The number of nodes in the input layer 50a of the first neural network 50 and the number of pixels of the input training data 75 correspond to the product of the number of brightness and hue contained in the image (for example, brightness 72y, first hue 72cb and second hue 72cr3 in the above example). The hue vector data 74 is input as a set 76 to the input layer 50a of the first neural network 50. The label value 77 of each pixel of the training data 75 is used as the output layer 50b of the first neural network to train the first neural network 50.

[0071] The first neural network 50 extracts feature quantities for the types of morphological cells and cell features described above based on the training data 75. The output layer 50b of the first neural network outputs a result reflecting the above feature quantities. Each result output from the flexible maximum function of the first neural network 50 is input to the input layer 51a of the second neural network 51. In addition, the cell morphology of cells belonging to a certain cell line is similar, so the deep learning algorithm is trained so that the deep learning algorithm 51 having the second neural network 51 is further specialized for identifying the types of specific morphological cells and the features of specific cells. Therefore, the label value 77 of the training data 75 is also input to the output layer of the second neural network.

[0072] Figure 2 Reference numerals 50c and 51c denote intermediate layers.

[0073] The first deep learning algorithm 60 having the first neural network 60 trained in this way and the second deep learning algorithm 61 having the second neural network 61 are combined and used as an identifier for identifying which of several cell types morphologically classified under a certain cell group the analysis target cell belongs to.

[0074] <Image analysis method>

[0075] Figure 4 An example of an image analysis method is shown. In the image analysis method, analysis data 81 is generated from an analysis image 78 obtained by photographing the analysis target cell. The analysis image 78 is an image obtained by photographing the analysis target cell. For example, the analysis image 78 can be acquired using a well-known imaging device such as an optical microscope or a virtual slide scanner. Figure 4In the example of , the analysis image 78 and the training image 70 are the same, the original image taken by Sysmex DI-60 at 360 pixels × 365 pixels is reduced to 255 pixels × 255 pixels, but it is not necessary to reduce it. The number of pixels of the analysis image 78 is not particularly limited as long as it can be analyzed, and it is preferably more than 100 pixels on one side. In addition, Figure 4 In the example, there are red blood cells around the segmented neutrophils, and the image can be cropped to retain only the target cells. An image can be used as the analysis image 78 if it contains at least one cell of the analysis target cell (which can contain red blood cells and normal-sized platelets) and the pixels belonging to the analysis target cell account for more than about 1 / 9 of the total pixels of the image.

[0076] As an example, the preferred technical solution is: In this embodiment, the camera is photographed using RGB colors and CMY colors. Preferably, the color image represents the depth or brightness of each primary color such as red, green and blue or cyan, magenta, and yellow with a 24-bit value (8 bits × 3 colors). The image 78 for analysis only needs to include at least one hue and the depth or brightness of the hue, and more preferably includes at least two hues and the depth or brightness of each hue. Information including the hue and the depth or brightness of the hue is also called color tone.

[0077] For example, convert from RGB color to a format that includes brightness information and hue information. The formats that include brightness information and hue information include YUV (YCbCr, YPbPr, YIQ, etc.). Here, the conversion to YCbCr format is explained as an example. Here, because the image used for analysis is RGB color, it is converted to brightness 79Y, the first hue (such as blue) 79Cb and the second hue (such as red) 79Cr. The conversion from RGB to YCbCr can be performed using well-known methods. For example, it can be converted from RGB to YCbCr in accordance with the international standard ITU-R BT. 601. For example Figure 4 As shown, the converted brightness 79Y, the first hue 79Cb and the second hue 79Cr can be represented by a matrix of gradation values ​​(hereinafter also referred to as a hue matrix 79y, 79cb, 79cr). The brightness 79Y, the first hue 79Cb and the second hue 79Cr are represented by 256 gradations from 0 to 255. Here, the image for analysis can be converted using the three primary colors of red R, green G and blue B and the three primary colors of cyan C, magenta M and yellow Y instead of the brightness 79Y, the first hue 79Cb and the second hue 79Cr.

[0078] Next, based on the hue matrix 79y, 79cb, 79cr, hue vector data 80 is generated for each pixel by combining three gradation values ​​of brightness 79y, first hue 79cb, and second hue 79cr. A collection of hue vector data 80 generated from one analysis image 78 is generated as analysis data 81.

[0079] It is preferable that the generation of the analysis data 81 and the generation of the training data 75 are the same in at least the imaging conditions and the generation conditions of the vector data input to the neural network from each image.

[0080] The analysis data 81 is input to the input layer 60a of the first neural network 60 constituting the trained first deep learning algorithm 60. The first deep learning algorithm extracts feature quantities from the analysis data 81, and outputs the results from the output layer 60c of the first neural network 60. The values ​​output from the output layer 60c are the probabilities of each of the classifications and features of the analysis target cells contained in the analysis image belonging to the morphological cell input as the training data.

[0081] Next, the result output from the output layer 60c is input to the input layer 61a of the second neural network 61 constituting the trained second deep learning algorithm 61. Based on the input feature quantity, the second deep learning algorithm 61 outputs the probability of the analysis target cell contained in the analysis image belonging to each of the classification and characteristics of the morphological cell input as training data from the output layer 61b. In addition, in the probability, it is determined that the analysis target cell contained in the analysis image belongs to the morphological classification with the largest value, and the label value associated with the type of morphological cell or the characteristic of the cell is output. The label value itself is output as the relevant data 83 of the cell morphology, or the label value is replaced with information (such as terms) indicating the presence or absence of the type of morphological cell or the characteristic of the cell as the relevant data 83 of the cell morphology. Figure 4 In the example, the identifier outputs the label value "1" as the most likely label value 82 based on the analyzed data 81, and outputs the text data "segmented neutrophil" corresponding to the label value as the cell morphology-related data 83.

[0082] Figure 4 Reference numerals 60c and 61c denote intermediate layers.

[0083] [2. Image analysis system 1]

[0084] <Structure of image analysis system 1>

[0085] A second embodiment of the present invention relates to an image analysis system.

[0086] Reference Figure 5The image analysis system involved in the second embodiment includes a deep learning device 100A and an image analysis device 200A. The manufacturer-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A uses training data to make the neural network 50 learn, and provides the deep learning algorithm 60 trained by the training data to the user. The deep learning algorithm composed of the neural network 60 that has completed learning is provided from the deep learning device 100A to the image analysis device 200A via the recording medium 98 or the network 99. The image analysis device 200A uses the deep learning algorithm composed of the neural network 60 that has completed learning to analyze the image to be analyzed.

[0087] The deep learning device 100A is composed of, for example, a general-purpose computer, and performs deep learning processing based on the flowchart described below. The image analysis device 200A is composed of, for example, a general-purpose computer, and performs image analysis processing based on the flowchart described below. The recording medium 98 is a computer-readable and non-temporary tangible recording medium such as a DVD-ROM or a USB memory.

[0088] The deep learning device 100A is connected to the imaging device 300. The imaging device 300 includes an image sensor 301 and a fluorescence microscope 302, and captures a bright field image of a training specimen 308 placed on a stage 309. The training specimen 308 is stained as described above. The deep learning device 100A acquires the training image 70 captured by the imaging device 300.

[0089] The image analysis device 200A is connected to the imaging device 400. The imaging device 400 includes an image sensor 401 and a fluorescence microscope 402, and captures a bright field image of an analysis target sample 408 placed on a stage 409. The analysis target sample 408 is previously stained as described above. The image analysis device 200A acquires the analysis target image 78 captured by the imaging device 400.

[0090] As the imaging devices 300 and 400 , a well-known optical microscope or virtual slide scanner or the like having a function of imaging a specimen can be used.

[0091] <Hardware structure of deep learning device>

[0092] Reference Figure 6 The manufacturer-side device 100 (deep learning device 100A, deep learning device 100B) includes a processing unit 10 ( 10A, 10B), an input unit 16 , and an output unit 17 .

[0093] The processing unit 10 includes a CPU (Central Processing Unit) 11 for performing data processing described later, a memory 12 for a work area used for data processing, a recording unit 13 for recording programs and processing data described later, a bus 14 for transmitting data between the units, an interface unit 15 for inputting and outputting data with external devices, and a GPU (Graphics Processing Unit) 19. An input unit 16 and an output unit 17 are connected to the processing unit 10. As an example, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator to assist the calculation processing (e.g., parallel calculation processing) performed by the CPU 11. That is, the processing performed by the CPU 11 in the following description also includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

[0094] In addition, the processing unit 10 performs the following Figure 8 The program and the neural network 50 before training according to the present invention are recorded in advance in the recording unit 13 in the form of an executable file, for example, according to the processing of each step described in the description. The executable file format is, for example, a format generated by converting a programming language by a compiler. The processing unit 10 uses the program recorded in the recording unit 13 to perform the training processing of the first neural network 50 and the second neural network 51 before training.

[0095] In the following description, unless otherwise specified, the processing performed by the processing unit 10 refers to the processing performed by the CPU 11 based on the program stored in the recording unit 13 or the memory 12 and the neural network 50. The CPU 11 temporarily stores necessary data (intermediate data during processing, etc.) using the memory 12 as a work area, and appropriately records data to be stored for a long time, such as calculation results, in the recording unit 13.

[0096] <Hardware structure of image analysis device>

[0097] Reference Figure 7 The user-side device 200 (image analysis device 200A, image analysis device 200B, image analysis device 200C) includes a processing unit 20 ( 20A, 20B, 20C), an input unit 26 , and an output unit 27 .

[0098] The processing unit 20 includes a CPU (Central Processing Unit) 21 for performing data processing described later, a memory 22 for a work area used for data processing, a recording unit 23 for recording programs and processing data described later, a bus 24 for transmitting data between the units, an interface unit 25 for inputting and outputting data with external devices, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. As an example, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator to assist the calculation processing (e.g., parallel calculation processing) performed by the CPU 21. That is, the processing performed by the CPU 21 in the following description also includes the processing performed by the CPU 21 using the GPU 29 as an accelerator.

[0099] In addition, the processing unit 20 records the program and the deep learning algorithm 60 of the trained neural network structure according to the present invention in advance in the recording unit 23 in the form of an executable file, for example, in order to perform the processing of each step described in the following image analysis processing. The executable file format is, for example, a format generated by converting from a programming language by a compiler. The processing unit 20 performs processing using the program recorded in the recording unit 23 and the first deep learning algorithm 60 and the second deep learning algorithm 61.

[0100] In the following description, unless otherwise specified, the processing performed by the processing unit 20 refers to the processing actually performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the recording unit 23 or the memory 22. The CPU 21 temporarily stores necessary data (intermediate data during processing, etc.) using the memory 22 as a work area, and appropriately records data to be stored for a long time, such as calculation results, in the recording unit 23.

[0101] <Functional modules and processing steps>

[0102] (Deep Learning Processing)

[0103] Reference Figure 8 The processing unit 10A of the deep learning device 100A according to this embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. A program that enables a computer to perform deep learning processing is installed in the recording unit 13 or the memory 12 of the processing unit 10A, and the CPU 11 executes the program to implement these functional modules. The training data database (DB) 104 and the algorithm database (DB) 105 are recorded in the recording unit 13 or the memory 12 of the processing unit 10A.

[0104] The imaging device 300 pre-images the training image 70 and pre-stores it in the recording unit 13 or the memory 12 of the processing unit 10A. The first deep learning algorithm 50 and the second deep learning algorithm 51 correspond to, for example, the morphological cell type and cell characteristics to which the analysis target cell belongs, and are pre-stored in the algorithm database 105.

[0105] The processing unit 10A of the deep learning device 100A performs Fig. 9 Processing as shown. Use Figure 8 To explain the functional blocks shown in FIG. 1 , the training data generation unit 101 performs the processing of steps S11, S12, S16, and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15.

[0106] use Fig. 9 An example of deep learning processing performed by the processing unit 10A will be described.

[0107] First, the processing unit 10A acquires the training image 70. The training image 70 is acquired from the imaging device 300 or the recording medium 98 by the operator's operation or through the I / F unit 15 via the network. When acquiring the training image 70, information related to which of the types of morphologically classified cells and / or morphological cell features the training image 70 represents is also acquired. The information related to which of the types of morphologically classified cells and / or morphological cell features the training image 70 represents may be associated with the training image 70, or may be input by the operator from the input unit 16.

[0108] In step S11 , the processing unit 10A converts the acquired training image 70 into the brightness Y, the first hue Cb, and the second hue Cr, and generates the hue vector data 74 according to the procedure described in the above-mentioned method of generating training data.

[0109] In step S12, the processing unit 10A gives a label value corresponding to the tone vector data 74 based on the following information: information related to the training image 70 indicating which of the types of morphologically classified cells and / or the characteristics of cells in the morphological classification is represented; and label values ​​associated with the types of morphologically classified cells or the characteristics of cells in the morphological classification stored in the memory 12 or the recording unit 13. In this way, the processing unit 10A generates the training data 75.

[0110] Fig. 9 In step S13 shown, the processing unit 10A trains the first neural network 50 and the second neural network 51 using the training data 75. Each time the training is performed using a plurality of training data 75, the training results of the first neural network 50 and the second neural network 51 are accumulated.

[0111] In the image analysis method according to the present embodiment, a convolutional neural network is used. Since the stochastic gradient descent method is used, in step S14, the processing unit 10A determines whether the training results of a predetermined number of trials have been accumulated. When the training results of a predetermined number of trials have been accumulated (yes), the processing unit 10A proceeds to the processing of step S15. When the training results corresponding to the predetermined number of trials have not been accumulated (no), the processing unit 10A proceeds to the processing of step S16.

[0112] Next, when the training results of a certain number of trials are accumulated, in step S15, the processing unit 10A uses the training results accumulated in step S13 to update the connection weights w of the first neural network 50 and the second neural network 51. In the image analysis method according to the present embodiment, since the stochastic gradient descent method is used, the connection weights w of the first neural network 50 and the second neural network 51 are updated after the learning results of a certain number of trials are accumulated. Specifically, the process of updating the connection weights w is the process of implementing the gradient descent method operation shown in (Formula 11) and (Formula 12) described later.

[0113] In step S16, the processing unit 10A determines whether the first neural network 50 and the second neural network 51 have been trained using the predetermined number of training data 75. If the training has been performed using the predetermined number of training data 75 (Yes), the deep learning process ends.

[0114] When the first neural network 50 and the second neural network 51 have not been trained using the predetermined number of training data 75 (No), the processing unit 10A proceeds from step S16 to step S17 and performs the processes from step S11 to step S16 on the next training image 70 .

[0115] According to the above-described process, the first neural network 50 and the second neural network 51 are trained to obtain the first deep learning algorithm 60 and the second deep learning algorithm 61 .

[0116] (Structure of Neural Network)

[0117] As described above, a convolutional neural network is used in this embodiment. Fig.10 (a) is an example of the structure of the first neural network 50 and the second neural network 51. The first neural network 50 and the second neural network 51 include input layers 50a, 51a, output layers 50b, 51b, and intermediate layers 50c, 51c between the input layers 50a, 51a and the output layers 50b, 51b, and the intermediate layers 50c, 51c are composed of a plurality of layers. The number of layers constituting the intermediate layers 50c, 51c may be, for example, 5 or more.

[0118] In the first neural network 50 and the second neural network 51, a plurality of nodes 89 arranged in layers are connected between layers. Thus, information is transmitted only in one direction indicated by arrow D in the figure from the input layer 50a, 51a to the output layer 50b, 51b.

[0119] (Calculation of each node)

[0120] Fig.10 (b) is a schematic diagram of the calculation of each node. Each node 89 receives several inputs and calculates one output (z). Fig.10 In the example of (b), the node 89 receives 4 inputs. The total input (u) received by the node 89 is expressed as follows (Equation 1).

[0121]

Number 1

[0122]

[0123] Each input is multiplied by a different weight. In (Equation 1), b is a value called bias. The output (z) of a node is the output of a certain function f of the total input (u) represented by (Equation 1), expressed as (Equation 2) below. Function f is called an activation function.

[0124]

Number 2

[0125]

[0126] Fig.10 (c) is a schematic diagram of the calculation between nodes. In the neural network 50, the nodes that output the result (z) represented by (Formula 2) for the total input (u) represented by (Formula 1) are arranged in layers. The output of the node in the previous layer is the input of the node in the next layer. Fig.10 In the example of (c), the output of node 89a of the layer on the left side of the figure is the input of node 89b of the layer on the right side of the figure. Each node 89b of the layer on the right side receives the output from node 89a of the layer on the left side. Each connection between each node 89a of the layer on the left side and each node 89b of the layer on the right side is multiplied by a different weight. If the outputs of several nodes 89a of the layer on the left side are x1 to x4, then the inputs to the three nodes 89b of the layer on the right side are respectively expressed by (Equation 3-1) to (Equation 3-3) below.

[0127]

Number 3

[0128]

[0129] The above (Equation 3-1) to (Equation 3-3) can be summarized as (Equation 3-4). Here, i=1, ...I, j=1, ...J.

[0130]

Number 4

[0131]

[0132] The output is obtained by applying (Equation 3-4) to the activation function. The output is expressed as follows (Equation 4).

[0133]

Number 5

[0134]

[0135] (Activation function)

[0136] In the image analysis method according to the embodiment, a rectified linear unit function is used as an activation function. The rectified linear unit function is expressed by the following (Formula 5).

[0137]

Number 6

[0138]

[0139] (Equation 5) is a function that makes the part where u<0 in the linear function of z=u become u=0. Fig. 9 In the example of (c), according to (Equation 5), the output of the node where j=1 is expressed by the following equation.

[0140]

Number 7

[0141]

[0142] (Learning of Neural Networks)

[0143] Suppose the function represented by the neural network is y(x:w), then if the parameter w of the neural network is changed, the function y(x:w) will also change. Adjusting the function y(x:w) so that the neural network selects a more appropriate parameter w for the input x is called learning of the neural network. Given an array, the input and output of the function represented by the neural network. When the preferred output for a certain input x is d, the given input and output group is {(x1, d1), (x2, d2), ..., (x n , d n )}. The set of groups represented by (x, d) is called training data. Specifically, Figure 2 The set of groups consisting of the color density value of each pixel in the monochrome image of R, G, and B and the label of the true value image is Figure 2 The training data shown. The learning of the neural network refers to: adjusting the weights w so that for all input-output groups (x n , d n ) for a given input x n The output of the neural network y(x n :w) Try to get as close to output dn The error function is a measure of how close the function represented by the neural network is to the training data.

[0144]

Number 8

[0145]

[0146] The error function is also called a loss function. The error function E(w) used in the image analysis method according to the embodiment is expressed by the following (Formula 6). (Formula 6) is called cross entropy.

[0147]

Number 9

[0148]

[0149] The calculation method of the cross entropy of (Equation 6) is explained. In the output layer 50b of the neural network 50 used in the image analysis method involved in the embodiment, that is, in the final layer of the neural network, an activation function is used to classify the input x into a finite number of levels according to the content. The activation function is called a softmax function and is represented by the following (Equation 7). In addition, the output layer 50b is arranged with the same number of nodes as the number of levels k. The total input u of each node k (k=1,...K) of the output layer L is the output u from the previous layer L-1 k （L） Given. Therefore, the output of the kth node in the output layer is expressed as follows (Formula 7).

[0150]

Number 10

[0151]

[0152] (Equation 7) is the flexible maximum function. The output y1,…,y determined by (Equation 7) K The sum of is always 1.

[0153] If each level is represented by C1, ..., C K , then the output y of node k in the output layer L is K (i.e. z ｋ （Ｌ） ) indicates that the given input x belongs to level C K The probability of . Refer to the following (Formula 8). The input x is classified into the class with the highest probability represented by (Formula 8).

[0154]

Number 11

[0155]

[0156] In neural network learning, the function represented by the neural network is regarded as a model of the posterior probability of each level. Under such a probability model, the likelihood of the weight w relative to the training data is evaluated, and the weight w that maximizes the likelihood is selected.

[0157] The target output d obtained by the flexible maximum function of (Equation 7) is only n is recorded as 1, and all other outputs are recorded as 0. n =[d n1 , …, d nK ]This vector form is expressed, for example, when the input x n When the correct level is C3, only the target output d n3 The target outputs other than 1 are recorded as 0. If encoded like this, the posterior distribution is expressed as follows (Equation 9).

[0158]

Number 12

[0159]

[0160] With the training data {(x n , d n The likelihood L(w) of the weight w relative to )} (n=1, ..., N) is expressed as follows (Equation 10). Taking the logarithm of the likelihood L(w) and reversing the sign, the error function of (Equation 6) is derived.

[0161]

Number 13

[0162]

[0163] Learning means minimizing the error function E(w) calculated based on the training data with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is represented by (Formula 6).

[0164] Minimizing the error function E(w) with respect to parameter w has the same meaning as finding the local minimum of the function E(w). Parameter w is the weight of the connection between nodes. The minimum of the weight w is found by iterative calculation, which takes an arbitrary initial value as the starting point and repeatedly updates the parameter w. An example of the above operation is the gradient descent method.

[0165] In the gradient descent method, a vector represented by the following (Equation 11) is used.

[0166]

Number 14

[0167]

[0168] In the gradient descent method, the process of moving the current parameter w in the negative gradient direction (i.e. -∇E) is repeated many times. If the current weight is w （t） , the weight after the move is w （t＋1） , the calculation by gradient descent is expressed as follows (Equation 12). The value t refers to the number of times the parameter w is moved.

[0169]

Number 15

[0170]

[0171] Label [number 16]

[0172]

[0173] is a constant that determines the update amount of parameter w, and is called the learning coefficient. By repeatedly performing the calculation represented by (Equation 12), the error function E(w) increases with the increase of the value t. （t） ) decreases, and the parameter w reaches a minimum.

[0174] In addition, the calculation of (Equation 12) can be performed for all training data (n=1, ..., N), or only for a part of the training data. The gradient descent method performed only for a part of the training data is called stochastic gradient descent. The stochastic gradient descent method is used in the image analysis method involved in the embodiment.

[0175] (Image analysis processing)

[0176] Fig.11 A functional block diagram of an image analysis device 200A that performs image analysis processing, until data 83 related to cell morphology is generated from an analysis target image 78. The processing unit 20A of the image analysis device 200A includes an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. The program for causing a computer to perform image analysis processing according to the present invention is installed in a recording unit 23 or a memory 22 of the processing unit 20A, and the CPU 21 executes the program to realize these functional blocks. A training data database (DB) 104 and an algorithm database (DB) 105 are provided from the deep learning device 100A via a recording medium 98 or a network 99, and recorded in the recording unit 23 or the memory 22 of the processing unit 20A.

[0177] The photographing device 400 photographs the analysis target image 78 and stores it in the recording unit 23 or the memory 22 of the processing unit 20A. The first deep learning algorithm 60 and the second deep learning algorithm 61 that have completed the training and contain the connection weight w are stored in the algorithm database 105, for example, in correspondence with the type of cells based on morphological classification to which the analysis target cell belongs and the characteristics of the cells, and function as a part of the program that enables the computer to perform image analysis processing, that is, a program module. That is, the first deep learning algorithm 60 and the second deep learning algorithm 61 are used in a computer with a CPU and a memory, and are used to: identify which of the types of several cells morphologically classified under a certain cell group the analysis target cell belongs to, and generate the relevant data 83 of the cell morphology. In addition, the generated data is output as needed. In addition, the CPU 21 of the processing unit 20A causes the computer to perform calculations or processing of unique information corresponding to the purpose of use. Specifically, the CPU 21 of the processing unit 20A uses the first deep learning algorithm 60 and the second deep learning algorithm 61 recorded in the recording unit 23 or the memory 22 to generate the relevant data 83 of the cell morphology. The CPU 21 of the processing unit 20A inputs the analysis data 81 to the input layer 60a, and outputs the feature value of the analysis image 78 calculated by the first deep learning algorithm 60 from the output layer 60b. The CPU 21 of the processing unit 20A inputs the feature value output from the first deep learning algorithm 60 to the input layer 61a of the second deep learning algorithm, and outputs the label value of the type of cell or the feature of the cell to which the recognized cell contained in the analysis target image belongs based on morphological classification from the output layer 61b.

[0178] use Fig.11 To explain the functional blocks shown in FIG. 2 , the analysis data generation unit 201 performs the processing of steps S21 and S22. The analysis data input unit 202 performs the processing of steps S23, S24, S25, and S27. The analysis unit 203 performs the processing of step S26.

[0179] use Fig.12 The example of the image analysis processing performed by the processing unit 20A will be described up to the point where the data 83 on the cell morphology is generated from the analysis target image 78 .

[0180] First, the processing unit 20A acquires the analysis image 78. The analysis image 78 is acquired from the imaging device 400 or the recording medium 98 by a user operation or through the I / F unit 25 via a network.

[0181] In step S21, Fig. 9 In step S11 shown, the acquired analysis image 78 is similarly converted into the brightness Y, the first hue Cb, and the second hue Cr, and the hue vector data 80 is generated according to the procedure described in the above-mentioned method of generating analysis data.

[0182] Next, in step S22 , the processing unit 20A generates analysis data 81 from the hue vector data 80 according to the procedure described in the above-mentioned method of generating analysis data.

[0183] Next, in step S23 , the processing unit 20A acquires the first deep learning algorithm and the second deep learning algorithm stored in the algorithm database 105 .

[0184] Next, in step S24, the processing unit 20A inputs the analysis data 81 to the first deep learning algorithm. The processing unit 20A inputs the feature quantity output from the first deep learning algorithm to the second deep learning algorithm according to the steps described in the above-mentioned image analysis method, and outputs the label value to which the analysis target cell contained in the analysis image is determined to belong from the second deep learning algorithm. The processing unit 20A stores the label value in the memory 22 or the recording unit 23.

[0185] In step S25, the processing unit 20A determines whether all the first acquired analysis images 78 have been recognized. If the recognition of all the analysis images 78 has been completed (yes), the process proceeds to step S26, and the analysis results including the cell morphology-related data 83 are output. If the recognition of all the analysis images 78 has not been completed (no), the process proceeds to step S27, and the processing from step S21 to step S25 is performed on the analysis images 78 that have not been recognized.

[0186] According to this embodiment, regardless of the skill of the examiner, the type of cell and the characteristics of the cell can be identified based on morphological classification, and the occurrence of inconsistency in morphological examination can be suppressed.

[0187] <Computer Program>

[0188] The present invention includes a computer program for image analysis that causes a computer to execute the processing of steps S11 to S17 and / or S21 to S27 and analyzes cell morphology.

[0189] In addition, one embodiment of the present invention relates to a program product such as a storage medium storing the computer program. That is, the computer program is stored in a storage medium such as a hard disk, a semiconductor storage element such as a flash memory, or an optical disk. There is no particular limitation on the storage format of the program stored in the storage medium as long as the prompting device can read the program. Preferably, the storage in the storage medium is non-volatile.

[0190] [3. Image analysis system 2]

[0191] <Structure of image analysis system 2>

[0192] Other technical solutions of the image analysis system are explained.

[0193] Fig.13 The structure example of the second image analysis system is shown. The second image analysis system includes a user-side device 200, and the user-side device 200 operates as a comprehensive image analysis device 200B. The image analysis device 200B is composed of, for example, a general-purpose computer, and performs the two processes of deep learning processing and image analysis processing described in the above-mentioned image analysis system 1. That is, the second image analysis system is an independently operated system that performs deep learning and image analysis on the user side. The comprehensive image analysis device 200B provided on the user side of the second image analysis system assumes the functions of both the deep learning device 100A and the image analysis device 200A involved in the first embodiment.

[0194] Fig.13 In FIG. 4 , the image analysis device 200B is connected to the imaging device 400. The imaging device 400 captures the training image 70 during the deep learning process, and captures the analysis target image 78 during the image analysis process.

[0195] <Hardware Structure>

[0196] The hardware structure of the image analysis device 200B and Figure 7 The hardware structure of the user-side devices 200 shown is the same.

[0197] <Functional modules and processing steps>

[0198] Fig.14The functional block diagram of the image analysis device 200B is shown. The processing unit 20B of the image analysis device 200B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. The program that causes the computer to execute the deep learning process and the image analysis process is installed in the recording unit 23 or the memory 22 of the processing unit 20B, and the CPU 21 executes the program to realize these functional blocks. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 23 or the memory 22 of the processing unit 20B, and are common to both the deep learning and the image analysis process. The first neural network 60 and the second neural network 61 that have been trained are stored in the algorithm database 105 in advance, for example, in correspondence with the type of cell to which the analysis target cell belongs based on morphological classification and the characteristics of the cell. The connection weight w is updated by the deep learning process, and the algorithm database 105 is stored as the deep learning algorithm 60. The training image 70 is captured in advance by the imaging device 400 and stored in advance in the training data database (DB) 104 or the recording unit 23 or the memory 22 of the processing unit 20B. The analysis target image 78 of the analysis target specimen is captured in advance by the imaging device 400 and stored in advance in the recording unit 23 or the memory 22 of the processing unit 20B.

[0199] The processing unit 20B of the image analysis device 200B performs deep learning processing. Fig. 9 The processing shown is performed during image analysis Fig.12 Processing as shown. Use Fig.13 In the explanation of each functional module shown, during deep learning processing, the training data generation unit 101 performs the processing of steps S11, S12, S16 and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15. During image analysis processing, the analysis data generation unit 201 performs the processing of steps S21 and S22. The analysis data input unit 202 performs the processing of steps S23, S24, S25 and S27. The analysis unit 203 performs the processing of step S26.

[0200] The steps of deep learning processing and image analysis processing performed by the image analysis device 200B are the same as those performed by the deep learning device 100A and the image analysis device 200A, respectively, except that the image analysis device 200B acquires the training image 70 from the imaging device 400 .

[0201] In the image analysis device 200B, the user can confirm the recognition accuracy of the recognizer. If the recognition result of the recognizer is different from the recognition result obtained by the user through image observation, the first deep learning algorithm and the second deep learning algorithm can be retrained using the analysis data 81 as the training data 75 and the recognition result obtained by the user through image observation as the label value 77. In this way, the training efficiency of the first neural network 50 and the second neural network 51 can be further improved.

[0202] [3. Image analysis system 3]

[0203] <Structure of image analysis system 3>

[0204] Other technical solutions of the image analysis system are explained.

[0205] Fig.15 The structure example of the third image analysis system is shown. The third image analysis system includes a manufacturer-side device 100 and a user-side device 200. The manufacturer-side device 100 operates as a comprehensive image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is composed of, for example, a general-purpose computer, and is a device on the cloud server side that performs the two types of processing, deep learning processing and image analysis processing, described in the above-mentioned image analysis system 1. The terminal device 200C is composed of, for example, a general-purpose computer, and is a terminal device on the user side that sends the analysis object image to the image analysis device 100B via the network 99 and receives the analysis result from the image analysis device 100B via the network 99.

[0206] The integrated image analysis device 100B provided on the manufacturer side of the third image analysis system assumes the functions of both the deep learning device 100A and the image analysis device 200A. On the other hand, the third image analysis system is provided with a terminal device 200C, which provides an input interface of the image 78 for analysis and an output interface of the analysis result to the terminal device 200C on the user side. That is, the third image analysis system is a cloud service type system in which the manufacturer side that performs deep learning processing and image analysis processing provides the image 78 for analysis to the user side through an input interface and provides the data 83 related to cell morphology to the user side through an output interface. The input interface and the output interface may be integrated.

[0207] The image analysis device 100B is connected to the imaging device 300 , and acquires the training image 70 captured by the imaging device 300 .

[0208] The terminal device 200C is connected to the imaging device 400 , and acquires the analysis target image 78 captured by the imaging device 400 .

[0209] <Hardware Structure>

[0210] The hardware structure of the image analysis device 100B is Figure 6 The hardware structure of the manufacturer-side device 100 shown in FIG. 200C is the same as that of the terminal device 200C. Figure 7 The hardware structure of the user-side devices 200 shown is the same.

[0211] <Functional modules and processing steps>

[0212] Fig.16 The functional block diagram of the image analysis device 100B is shown. The processing unit 10B of the image analysis device 100B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. The program that causes the computer to execute deep learning processing and image analysis processing is installed in the recording unit 13 or the memory 12 of the processing unit 10B, and the CPU 11 executes the program to realize these functional blocks. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 13 or the memory 12 of the processing unit 10B, and are common to both deep learning and image analysis processing. The first neural network 50 and the second neural network 51 are stored in the algorithm database 105 in advance, for example, in correspondence with the type of cell to which the analysis target cell belongs based on morphological classification and the characteristics of the cell. The connection weight w is updated by the deep learning processing and stored as the first deep learning algorithm 60 and the second deep learning algorithm 61.

[0213] The training image 70 is captured in advance by the imaging device 300 and recorded in advance in the training data database (DB) 104 or the recording unit 13 or the memory 12 of the processing unit 10B. The analysis target image 78 is captured in advance by the imaging device 400 and recorded in advance in the recording unit 23 or the memory 22 of the processing unit 20C of the terminal device 200C.

[0214] The processing unit 10B of the image analysis device 100B performs deep learning processing. Fig. 9 The processing shown is performed during image analysis Fig.12 Processing as shown. Use Fig.16 In the explanation of each functional module shown, during deep learning processing, the training data generation unit 101 performs the processing of steps S11, S12, S16 and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15. During image analysis processing, the analysis data generation unit 201 performs the processing of steps S21 and S22. The analysis data input unit 202 performs the processing of steps S23, S24, S25 and S27. The analysis unit 203 performs the processing of step S26.

[0215] The procedure of the deep learning process and the procedure of the image analysis process performed by the image analysis device 100B are the same as the procedures respectively performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment.

[0216] The processing unit 10B receives the analysis target image 78 from the terminal device 200C on the user side, and Fig. 9 Steps S11 to S17 shown generate training data 75 .

[0217] Fig.12 In step S26 shown, the processing unit 10B transmits the analysis result including the cell morphology-related data 83 to the user-side terminal device 200C. In the user-side terminal device 200C, the processing unit 20C outputs the received analysis result to the output unit 27.

[0218] As described above, the user of the terminal device 200C can obtain the data 83 on the cell morphology as the analysis result by sending the analysis target image 78 to the image analysis device 100B.

[0219] According to the image analysis device 100B according to the third embodiment, the user can use the identifier without acquiring the training data database 104 and the algorithm database 105 from the deep learning device 100A. This makes it possible to provide a service for identifying cell types and cell features based on morphological classification as a cloud service.

[0220] [4. Other implementation methods]

[0221] As mentioned above, although this invention was demonstrated using the summary and specific embodiment, this invention is not limited to the above summary and each embodiment.

[0222] The present invention illustrates a method of converting hue into brightness Y, first hue Cb, and second hue Cr to generate training data 75, but the conversion of hue is not limited to this. The hue may not be converted, for example, the three primary colors of red (R), green (G), and blue (B) may be used directly. Alternatively, two primary colors of any hue may be subtracted from the primary colors. Alternatively, only one of the three primary colors of red (R), green (G), and blue (B) (such as green (G)), that is, one primary color, may be used. It may also be converted into three primary colors of cyan (C), magenta (M), and yellow (Y). For example, the analyzed object image 78 is not limited to a color image of the three primary colors of red (R), green (G), and blue (B), but may be a color image of two primary colors, as long as it is an image containing more than one primary color.

[0223] In the above-mentioned training data generation method and analysis data generation method, in step S11, the processing units 10A, 20B, 10B generate the hue matrix 72y, 72cb, 72cr from the training image 70, but the training image 70 may be an image that has been converted into the brightness Y, the first hue Cb, and the second hue Cr. That is, the processing units 10A, 20B, 10B may directly obtain the brightness Y, the first hue Cb, and the second hue Cr from a virtual slide scanner, etc. at the beginning. Similarly, in step S21, the processing units 20A, 20B, 10B generate the hue matrix 79y, 79cb, 79cr from the analysis target image 78, but the processing units 20A, 20B, 10B may directly obtain the brightness Y, the first hue Cb, and the second hue Cr from a virtual slide scanner, etc. at the beginning.

[0224] In addition to RGB and CMY, YUV and CIE L ＊ a ＊ b ＊ Various color gamuts are used for image acquisition and tone conversion.

[0225] In the hue vector data 74 and the hue vector data 80, hue information is stored in the order of brightness Y, first hue Cb, and second hue Cr for each pixel, but the storage order and processing order of the hue information are not limited to this. However, it is preferred that the arrangement order of the hue information in the hue vector data 74 and the arrangement order of the hue information in the hue vector data 80 be the same.

[0226] In each image analysis system, the processing units 10A and 10B are an integrated device, but the processing units 10A and 10B do not necessarily need to be designed as an integrated device. They can be designed such that the CPU 11, the memory 12, the recording unit 13, the GPU 19, etc. are arranged in other places and connected by a network. The processing units 10A and 10B, the input unit 16, and the output unit 17 do not necessarily need to be arranged in one place. They can be designed such that they are arranged in different places and connected to each other by a network and can communicate. The processing units 20A, 20B, and 20C are also the same as the processing units 10A and 10B.

[0227] In the first to third embodiments described above, the functional modules of the training data generation unit 101, the training data input unit 102, the algorithm update unit 103, the analysis data generation unit 201, the analysis data input unit 202, and the analysis unit 203 are executed in a single CPU 11 or a single CPU 21, but these functional modules do not necessarily have to be executed in a single CPU, and can also be executed in a distributed manner on several CPUs. In addition, these functional modules can be executed in a distributed manner on several GPUs, or in a distributed manner on several CPUs and several GPUs.

[0228] In the second and third embodiments, the Fig. 9 and Fig.12 The processing program of each step described is recorded in advance in the recording unit 13, 23. This scheme may be replaced by the following scheme: the program is installed in the processing unit 10B, 20B from a computer-readable and non-temporary tangible recording medium 98 such as a DVD-ROM or USB memory. Alternatively, it can be designed that the processing unit 10B, 20B is connected to the network 99, and the program is downloaded from an external server (not shown) via the network 99 and installed.

[0229] In each image analysis system, the input unit 16, 26 is an input device such as a keyboard or a mouse, and the output unit 17, 27 is a display device such as a liquid crystal display. However, an alternative technical solution may be adopted, that is, the input unit 16, 26 and the output unit 17, 27 may be integrated and designed as a touch screen type display device. Alternatively, the output unit 17, 27 may be configured as a printer or the like.

[0230] In each of the above-mentioned image analysis systems, the camera 300 is directly connected to the deep learning device 100A or the image analysis device 100B, but the camera 300 may be connected to the deep learning device 100A or the image analysis device 100B via the network 99. The same is true for the camera 400. The camera 400 is directly connected to the image analysis device 200A or the image analysis device 200B, but the camera 400 may be connected to the image analysis device 200A or the image analysis device 200B via the network 99.

[0231] [5. Effects of Deep Learning Algorithms]

[0232] In order to verify the effect of the deep learning algorithm, the cell recognition accuracy of the cell recognition method using the existing machine learning method and the cell recognition method using the deep learning algorithm of the present invention was compared.

[0233] Peripheral blood smears were prepared using the smear preparation device SP-1000i, and cells were photographed using the blood imaging automatic analysis device DI-60. The staining was Giemsa staining.

[0234] The blood imaging automatic analysis device DI-60 was used to perform cell identification using the existing machine learning method. In addition, three people including a doctor and an experienced technician observed the images and verified them.

[0235] Fig.17 This is the comparison result of blood cell classification accuracy. When using the deep learning algorithm, it is possible to make higher-precision judgments than the existing method.

[0236] Next, we investigated whether the deep learning algorithm of the present invention can identify morphological features present in myelodysplastic syndrome (MDS). Fig.18 shown.

[0237] like Fig.18 As shown, abnormal nuclear morphology, cavitation, and abnormal particle distribution can be identified with high accuracy.

[0238] As described above, the deep learning algorithm of the present invention can identify the types of cells and the characteristics of cells based on morphological classification with high accuracy.

[0239] Explanation of symbols

[0240] 200 Image analysis device

[0241] 10. Processing Department

[0242] 50,51 Pre-training deep learning algorithms

[0243] 50a, 51a Input layer

[0244] 50b, 51b output layer

[0245] 60,61 Completed training of deep learning algorithms

[0246] 75 Training Data

[0247] 81 Parsing Data

[0248] 83 Cell morphology data

Claims

1. An image analysis method, comprising the following steps: Inputting analysis data into a deep learning algorithm having a neural network structure, wherein the analysis data is generated from an image of an analysis target cell and includes information related to the analysis target cell, wherein the analysis target cell is a cell in peripheral blood; Calculating the probability of each of the morphological classifications of the analysis target cell belonging to a certain cell group by the deep learning algorithm; and Outputting data corresponding to the morphological classification whose calculated probability is the largest among the calculated probabilities of the morphological classifications, wherein, The certain cell population is a blood cell population, The morphological classification indicates abnormal appearance of the cells, and The abnormal manifestations of the cells include abnormal nuclear morphology in mother cells, vacuoles in mother cells, granules in mother cells, giant band-shaped neutrophils, abnormal nuclear morphology in band-shaped neutrophils, abnormal granule distribution in band-shaped neutrophils, giant segmented neutrophils, abnormal nuclear morphology in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hypersegmented nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

2. The image analysis method according to claim 1 further comprises the following steps: The morphological class to which the analysis target cell belongs is identified based on the calculated probability.

3. The image analysis method according to claim 1, characterized in that: The certain cell population is a population of cells belonging to a certain cell line.

4. The image analysis method according to claim 3, characterized in that: The certain cell line is the hematopoietic system.

5. The image analysis method according to claim 1, characterized in that: Each morphological classification indicates the type of cell.

6. The image analysis method according to claim 1, characterized in that: The plurality of cells include a plurality of cells selected from the group consisting of segmented neutrophils, band neutrophils, metamyelocytes, myelocytes, blasts, lymphocytes, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts, giant platelets, platelet aggregation, and megakaryocytes.

7. The image analysis method according to claim 1, characterized in that: The morphological classification of the plurality of cells comprises at least one selected from the group consisting of abnormal nuclear morphology, the presence of vacuoles, abnormal granule morphology, abnormal granule distribution, the presence of abnormal granules, abnormal cell size, the presence of inclusion bodies, and naked nuclei.

8. The image analysis method according to claim 1, characterized in that: The deep learning algorithm comprises: A first algorithm for calculating the probability of each item of the first morphological classification of the plurality of cells belonging to a certain cell group that the analysis target cell belongs to; The second algorithm is used to calculate the probability of each of the second morphological classifications of the plurality of cells belonging to a certain cell group that the analysis target cell belongs to.

9. The image analysis method according to claim 8, characterized in that: Each first morphological classification indicates the type of cell, Each second morphological category indicates abnormal behavior of the cells.

10. The image analysis method according to claim 1, characterized in that: The analysis data is generated from the captured images of stained blood cells.

11. The image analysis method according to claim 10, characterized in that: The stain is selected from Wright's stain, Giemsa stain, Swiss Giemsa stain and Mei Giemsa stain.

12. The image analysis method according to claim 1, characterized in that: The analysis data includes information related to brightness of the image of the analysis target cell and information related to at least two hues.

13. An image analysis device, comprising: Processing unit, for: Inputting analysis data into a deep learning algorithm having a neural network structure, wherein the analysis data is generated from an image of an analysis target cell and includes relevant information of the analysis target cell, wherein the analysis target cell is a cell in peripheral blood; Calculating the probability of each of the morphological classifications of the analysis target cell belonging to a certain cell group by the deep learning algorithm; and Outputting data corresponding to the morphological classification whose calculated probability is the largest among the calculated probabilities of the morphological classifications, wherein, The certain cell population is a blood cell population, The morphological classification indicates abnormal appearance of the cells, and The abnormal manifestations of the cells include abnormal nuclear morphology in mother cells, vacuoles in mother cells, granules in mother cells, giant band-shaped neutrophils, abnormal nuclear morphology in band-shaped neutrophils, abnormal granule distribution in band-shaped neutrophils, giant segmented neutrophils, abnormal nuclear morphology in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hypersegmented nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

14. The image analysis device according to claim 13, characterized in that: Each morphological classification indicates the type of cell.

15. A storage medium, the storage medium being a non-transitory computer-readable storage medium storing a program executed by a processor, characterized in that: Inputting analysis data into a deep learning algorithm having a neural network structure, wherein the analysis data is generated from an image of an analysis target cell and includes relevant information of the analysis target cell, wherein the analysis target cell is a cell in peripheral blood; Calculating the probability of each item of the morphological classification of the analysis target cell belonging to a certain cell group by the deep learning algorithm; and Outputting data corresponding to the morphological classification whose calculated probability is the largest among the calculated probabilities of the morphological classifications, wherein, The certain cell population is a blood cell population, The morphological classification indicates abnormal appearance of the cells, and The abnormal manifestations of the cells include abnormal nuclear morphology in mother cells, vacuoles in mother cells, granules in mother cells, giant band-shaped neutrophils, abnormal nuclear morphology in band-shaped neutrophils, abnormal granule distribution in band-shaped neutrophils, giant segmented neutrophils, abnormal nuclear morphology in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hypersegmented nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

Images