Image analysis method and device, storage medium and deep learning algorithm generation method

Abstract

The invention relates to an image analysis method and device, a storage medium and a deep learning algorithm generation method. The purpose of the present invention is to provide an image analysis method for recognizing the morphology of each of a plurality of cells included in an image for analysis with higher accuracy. An image analysis method for analyzing the morphology of a cell using a deep learning algorithm having a neural network structure inputs analysis data including information relating to a cell to be analyzed to the deep learning algorithm (60, 61), and analyzes the morphology of the cell using the deep learning algorithm (60, 61). 61) calculates the probability that the cell to be analyzed belongs to each of the morphological classifications of a plurality of cells belonging to the specific cell population, and analyzes the image by this method.

Description

[0001] This divisional application is a divisional application of a Chinese patent application with the application number 201910387008.0, the filing date of May 10, 2019, and the invention title of "Image Analysis Method, Device, Storage Medium, and Deep Learning Algorithm Generation Method". Technical Field

[0002] The present invention relates to an image analysis method, device, storage medium, and deep learning algorithm generation method for analyzing cell morphology. Background Art

[0003] Patent Document 1 discloses a cell recognition system for processing microscope images. In the cell recognition system, a model trained using a machine learning technique associates pixels in the acquired image with one or more of cells, cell boundaries, backgrounds, and their equivalents. The machine learning technique used is the random forest decision tree technique.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-534709. Summary of the Invention

[0007] Technical Problems to be Solved by the Invention

[0008] In cell examination, usually, an examiner observes cells through a microscope and morphologically identifies the types and characteristics of the cells. However, cells of the same lineage have similar morphologies. Therefore, in order to morphologically identify cells, it is necessary to observe many cell specimens and improve the recognition skills. In particular, proficiency is required when identifying abnormal cells that appear during illness. For example, in cases where the frequency of appearance of abnormal cells is low, such as in early myelodysplastic syndrome, there is a risk that an examiner with insufficient skills may overlook the abnormal cells.

[0009] In addition, the number of specimens that an examiner can observe in a day is limited, and observing more than 100 specimens in a day will also impose a burden on the examiner.

[0010] Flow-through type automatic blood cell classification devices, etc. can also increase the number of cell examination processes. However, the information obtained from flow-through type automatic blood cell classification devices is limited, and it is difficult to identify blood cells with a low frequency of appearance, such as blast cells, promyelocytes, and giant platelets.

[0011] As shown in the method described in Patent Document 1, a method of identifying cells using machine training techniques (also called machine learning) is also known. However, it is necessary for the user to create training data for training the machine learning model, and a huge amount of labor is required to generate the model. In addition, at present, since the training data is created by the user, the number of training data that can be created is limited, and there are still problems with the analysis accuracy and generalization performance of the machine learning model.

[0012] In addition, the method described in Patent Document 1 is a method of identifying cell parts and non-cell parts in a microscope image. Therefore, it is impossible to identify the type of each cell and what kind of abnormal manifestations it contains.

[0013] An object of the present invention is to provide an image analysis method for more accurately identifying the respective morphologies of several cells contained in an analysis image.

[0014] Technical means for solving technical problems

[0015] An embodiment of the present invention relates to an image analysis method. In the image analysis method, analysis data (80) is input into a deep learning algorithm (60, 61) having a neural network structure. The analysis data is generated from an image of an analysis target cell and contains information related to the analysis target cell; the probability of each item in the morphological classification of several cells belonging to a certain cell group to which the analysis target cell belongs is calculated by the deep learning algorithm. Through this embodiment, even if the examiner does not perform microscopic observation, the probability of each item in the morphological classification of several cells belonging to a certain cell group to which the analysis target cell belongs can be obtained and the image can be analyzed.

[0016] A preferred technical solution is: the image analysis method identifies the morphological classification to which the analysis target cell belongs based on the calculated probability. Through this embodiment, even if the examiner does not perform microscopic observation, it is possible to identify which morphological classification the analysis target cell matches.

[0017] A preferred technical solution is: the certain cell group is a group of blood cells. Through this embodiment, even if the examiner does not perform microscopic observation, the morphological classification of blood cells can be performed.

[0018] A preferred technical solution is: the certain cell group is a group of cells belonging to a certain cell line. A more preferred technical solution is: the certain cell line is the hematopoietic system. Through this embodiment, even if the examiner does not perform microscopic observation, the morphological classification of cells belonging to the same cell line can be performed.

[0019] The preferred technical solution is that each morphological classification indicates the type of cell. A more preferred technical solution is that the morphological classification includes the following: neutrophils including segmented neutrophils and banded neutrophils, metamyelocytes, myelocytes, promyelocytes, blast cells, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts, giant platelets, platelet aggregates, and megakaryocytes. According to this embodiment, cells of the same series with similar morphologies can also be identified.

[0020] The preferred technical solution is that each morphological classification indicates the abnormal manifestations of the cells. A more preferred technical solution is that the morphological classification includes at least one cell selected from the group consisting of the following items: abnormal nuclear morphology, presence of vacuoles, abnormal granule morphology, abnormal granule distribution, presence of abnormal granules, abnormal cell size, presence of inclusions, and naked nuclei. According to this embodiment, cells presenting abnormal manifestations can also be identified.

[0021] In the said embodiment, the relevant data of the morphology of the cells are the relevant data of the types of cells in the morphological classification and the relevant data of the characteristics of the cells in the morphological classification. Through this embodiment, the types and characteristics of morphological cells can be output.

[0022] The preferred technical solution is that in the said embodiment, the deep learning algorithm includes: a first algorithm that calculates the probability of each item in the first morphological classification of the analyzed cell belonging to several cells under a certain cell group; a second algorithm that calculates the probability of each item in the second morphological classification of the analyzed cell belonging to several cells under a certain cell group. For example, the first morphological classification is the type of the analyzed cell, and the second morphological classification is the abnormal manifestation of the analyzed cell. In this way, the recognition accuracy of cells with similar morphologies can be further improved.

[0023] In the said embodiment, the analysis data (80) is generated from an image of the stained blood cells taken. A more preferred technical solution is that the staining is selected from Wright staining, Giemsa staining, Swiss Giemsa staining, and May-Grünwald Giemsa staining. In this way, the same recognition as the microscopic observation carried out all the time can be performed.

[0024] The analysis data (80) and the training data (75) include the relevant information of the brightness of the analyzed object image and the training image and the relevant information of at least two hues. In this way, the recognition accuracy can be improved.

[0025] Other embodiments of the present invention relate to an image analysis device (200). The image analysis device (200) includes: a processing unit (10) that inputs analysis data (80) into a deep learning algorithm (60, 61) having a neural network structure. The analysis data is generated from an image of an analysis target cell and includes information related to the analysis target cell. The probability of each item in the morphological classification of the analysis target cell belonging to several cells under a certain cell group is calculated by the deep learning algorithm. A preferred technical solution is that each morphological classification is the identification of the type of the analysis target cell. Another preferred technical solution is that each morphological classification is the identification of the abnormal manifestation of the analysis target cell.

[0026] Other embodiments of the present invention relate to a storage medium, which is a non-transitory computer-readable storage medium storing a program executed by a processor. The program causes a computer (200) to perform the following processing: input analysis data (80) into a deep learning algorithm (60, 61) having a neural network structure. The analysis data (80) is generated from an image of an analysis target cell and includes information related to the analysis target cell. The probability of each item in the morphological classification of the analysis target cell belonging to several cells under a certain cell group is calculated by the deep learning algorithm.

[0027] Other embodiments of the present invention relate to a method for generating a trained deep learning algorithm (60, 61). In this embodiment, training data containing cell-related information is input into the input layers (50a, 51a) of neural networks (50, 51), and a label value is input as the output layer (50b, 51b). The label value is associated with each morphological classification of several cells under a certain cell group. A preferred technical solution is that each morphological classification indicates the type of the cell or the abnormal manifestation of the cell.

[0028] By using the image analysis device (200) and the trained deep learning algorithm (60, 61), the type and characteristics of morphological cells can be identified without being affected by the skills of the examiner.

[0029] Advantages of the Invention

[0030] The morphology of each of several cells contained in the analysis image can be identified. As a result, cell inspection can be performed without being affected by the skills of the examiner. Brief Description of the Drawings

[0031] Figure 1 A diagram showing the outline of the present invention;

[0032] Figure 2 A schematic diagram of an example of the generation step of training data, the training steps of the first deep learning algorithm, and the second deep learning algorithm;

[0033] Figure 3 Examples of label values;

[0034] Figure 4 Schematic diagram of an example of the generation step of the analysis data and the cell recognition step using a deep learning algorithm;

[0035] Figure 5 Schematic diagram of the structure example of the image analysis system 1;

[0036] Figure 6 Block diagram showing an example of the hardware structure of the manufacturer-side device 100;

[0037] Figure 7 Block diagram showing an example of the hardware structure of the user-side device 200;

[0038] Figure 8 Block diagram for explaining an example of the function of the deep learning device 100A;

[0039] Figure 9 Flowchart showing an example of the deep learning process;

[0040] Figure 10 Schematic diagram for explaining a neural network;

[0041] Figure 11 Block diagram for explaining an example of the function of the image analysis device 200A;

[0042] Figure 12 Flowchart showing an example of the image analysis process;

[0043] Figure 13 Schematic diagram of the structure example of the image analysis system 2;

[0044] Figure 14 Block diagram for explaining an example of the function of the comprehensive image analysis device 200B;

[0045] Figure 15 Schematic diagram of the structure example of the image analysis system 3;

[0046] Figure 16 Block diagram for explaining an example of the function of the comprehensive image analysis device 100B;

[0047] Figure 17 Shows the recognition result of the cell type obtained using the deep learning algorithm;

[0048] Figure 18 Shows the recognition result of the cell characteristics obtained using the deep learning algorithm. Detailed implementation

[0049] The outline and embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In addition, in the following description and drawings, the same reference numerals denote the same or similar components, and thus the description of the same or similar components will be omitted.

[0050] [1. Image analysis method]

[0051] The first embodiment of the present invention relates to an image analysis method for analyzing the morphology of cells. The image analysis method inputs analysis data including information related to cells to be analyzed into a classifier including a deep learning algorithm having a neural network structure. The classifier calculates the probability that the cells to be analyzed belong to each of several morphological classifications belonging to a certain cell group. Preferably, the image analysis method further includes the following process: identifying which of the several morphological classifications belonging to a certain cell group the cells to be analyzed belong to based on the probability.

[0052] In the first embodiment, the cells to be analyzed belong to a certain cell group. The certain cell group is a cell group constituting each organ in the body of mammals and birds. Normally, the certain cell group contains several cell types morphologically classified by histological microscopy and cytological microscopy. The morphological classification (also called "morphological classification") includes the classification of cell types and the classification of morphological characteristics of cells. Preferably, the cells to be analyzed are a cell group belonging to a certain cell line belonging to a certain cell group. A certain cell line refers to a cell group of the same lineage differentiated from a certain tissue stem cell. The certain cell line is preferably the hematopoietic system, and more preferably cells in the blood (also called "blood cells").

[0053] Existing methods classify hematopoietic system cells morphologically by a person observing a specimen stained for bright field microscopy under the bright field of a microscope. Preferably, the staining is selected from Wright staining, Giemsa staining, Swiss Giemsa staining, and May-Grünwald Giemsa staining. More preferably, it is May-Grünwald Giemsa staining. The specimen is not particularly limited as long as it satisfies the condition that the morphology of each cell belonging to a certain cell group can be observed individually. For example, smear specimens and imprints can be cited. A smear specimen using peripheral blood or bone marrow as a sample is preferred, and a smear specimen of peripheral blood is more preferred.

[0054] In morphological classification, blood cells include the following types of cells: neutrophils (including segmented neutrophils and banded neutrophils), metamyelocytes, myelocytes, promyelocytes, blast cells, lymphocytes, plasma cells, atypical lymphocytes, monocytes, eosinophils, basophils, erythroblasts (nucleated red blood cells, including pronormoblasts, basophilic normoblasts, polychromatophilic normoblasts, orthochromatic normoblasts, pro megaloblasts, basophilic megaloblasts, polychromatophilic megaloblasts, and orthochromatic megaloblasts), giant platelets, platelet aggregates, and megakaryocytes (nucleated megakaryocytes, including micromegakaryocytes), etc.

[0055] In addition, the certain cell population may include abnormal cells showing morphological abnormalities in addition to normal cells. The abnormality is manifested as the cell characteristics classified morphologically. Examples of abnormal cells are cells that appear when suffering from a certain disease, such as tumor cells, etc. In the hematopoietic system, a certain disease is a disease selected from the group consisting of myelodysplastic syndromes, leukemias (including acute myeloid leukemia, acute myeloid leukemia (with neutrophil differentiation), acute promyelocytic leukemia, acute myelomonocytic leukemia, acute monocytic leukemia, erythroleukemia, acute megakaryoblastic leukemia, acute myelogenous leukemia, acute lymphoblastic leukemia, lymphoblastic leukemia, chronic myelogenous leukemia, and chronic lymphocytic leukemia, etc.), malignant lymphomas (such as Hodgkin lymphoma and non-Hodgkin lymphoma, etc.), and multiple myeloma. In addition, in the hematopoietic system, cells with abnormal manifestations are cells having at least one morphological feature selected from the group consisting of the following items: abnormal nuclear morphology, presence of vacuoles, abnormal granule morphology, abnormal granule distribution, presence of abnormal granules, abnormal cell size, presence of inclusions, and naked nuclei.

[0056] Abnormal nuclear morphology can include: nuclear shrinkage; nuclear enlargement; excessive nuclear lobulation; the nucleus that should be lobulated normally but is not lobulated (including pseudo-Pelger nuclear abnormalities, etc.); presence of vacuoles; enlarged nucleoli; nuclear notches; the abnormal situation where one cell should have one nucleus but has two nuclei, etc.

[0057] Abnormalities in the overall morphology of the cell can include: presence of vacuoles in the cytoplasm (also called vacuolar degeneration); abnormal morphology of granules such as azurophilic granules, neutrophilic granules, eosinophilic granules, and basophilic granules; abnormal distribution of the granules (excessive, reduced, or disappeared); presence of abnormal granules (such as toxic granules, etc.); abnormal cell size (larger or smaller than normal); presence of inclusions (such as Dohle bodies and rod-shaped bodies, etc.); naked nuclei, etc.

[0058] <Summary of the Image Analysis Method>

[0059] Use Figure 1 To describe the summary of the image analysis method.

[0060] The recognizer used in the image analysis method includes several deep learning algorithms (sometimes simply called "algorithms") with a neural network structure. The preferred technical solution is: including a first deep learning algorithm 50 and a second deep learning algorithm 51. The first deep learning algorithm 50 extracts the feature quantity of cells, and the second deep learning algorithm 51 identifies the analysis target cells based on the feature quantity extracted by the first deep learning algorithm. A more preferred technical solution is: as Figure 1 shown, in addition to including the second deep learning algorithm downstream of the first deep learning algorithm, the recognizer can also include several deep learning algorithms (sometimes numbered as the second, third, fourth, fifth... the i-th) trained according to the recognition purpose. For example, the second deep learning algorithm identifies the types of cells classified based on morphology as described above. Additionally, for example, the second deep learning algorithm identifies the features of the cells classified based on morphology for each feature respectively. The preferred technical solution is: the first deep learning algorithm is a convolutional neural network, and the deep learning algorithms after the second located downstream of the first deep learning algorithm are fully connected neural networks.

[0061] Next, the generation method of the training data 75 and the image analysis method will be described using Figures 2 to 4 an example. For ease of explanation, the first deep learning algorithm and the second deep learning algorithm will be used for the following description.

[0062] <Generation of Training Data>

[0063] The training image 70 used for training the deep learning algorithm is an image obtained by photographing cells whose known types and features of cells classified based on morphology correspond to the analysis target cells. It is preferred to use a specimen containing cells of the same type as the analysis target cells and prepare the specimen for photographing the training image 70 through the same specimen preparation method and staining method as the specimen containing the analysis target cells. Additionally, it is preferred to photograph the training image 70 under the same shooting conditions as the analysis target cells.

[0064] For example, well-known photographing devices such as optical microscopes or virtual slide scanners can be used to obtain the training image 70 for each cell in advance. Figure 2 In the example of Figure 2In the example, red blood cells are present around the neutrophil as the center, and the image can also be cropped to retain only the target cells. In one image, there should be at least 1 cell of the training target cells (red blood cells and normally sized platelets can be included), and if the pixels belonging to the training target cells account for about 1 / 9 or more of the total pixels of the image, it can be used as the training image 70.

[0065] As an example, a preferred technical solution is: in this embodiment, the shooting device shoots with RGB colors and CMY colors, etc. Preferably, in a color image, the shades or brightness of each primary color such as red, green, and blue or cyan, magenta, and yellow are represented by 24-bit values (8 bits × 3 colors). The training image 70 only needs to contain at least 1 hue and the shade or brightness of this hue, and more preferably contains at least 2 hues and the shade or brightness of each hue. The information containing the hue and the shade or brightness of this hue is also called the color tone.

[0066] Next, convert the color tone information of each pixel, for example, from RGB color to a format including brightness information and hue information. Formats including brightness information and hue information can include YUV (such as YCbCr, YPbPr, YIQ, etc.). Here, the conversion to the YCbCr format is taken as an example for explanation. Here, since the training image is in RGB color, it is converted to brightness 72Y, the first hue (such as the blue system) 72Cb, and the second hue (such as the red system) 72Cr. It is possible to convert from RGB to YCbCr using well-known methods. For example, it is possible to convert from RGB to YCbCr in accordance with the international standard ITU-R BT.601. As Figure 2 shown, the converted brightness 72Y, the first hue 72Cb, and the second hue 72Cr can be respectively represented by a row and column of hierarchical values (hereinafter also called the color tone row and column 72y, 72cb, 72cr). The brightness 72Y, the first hue 72Cb, and the second hue 72Cr are respectively represented by 256 levels from 0 to 255 levels. Here, it is also possible to convert the training image using the three primary colors of red R, green G, and blue B, or the three primary colors of cyan C, magenta M, and yellow Y instead of the brightness 72Y, the first hue 72Cb, and the second hue 72Cr.

[0067] Next, based on the color tone row and column 72y, 72cb, 72cr, generate color tone vector data 74 by combining the three hierarchical values of brightness 72y, the first hue 72cb, and the second hue 72cr for each pixel.

[0068] Next, for example, since Figure 2 the training image 70 is a segmented neutrophil, each color tone vector data 74 generated from Figure 2 the training image 70 is assigned a label value 77 of "1" to represent the segmented neutrophil, obtaining the training data 75. Figure 2In this case, for convenience, the training data 75 is represented by 3 pixels × 3 pixels, but in fact, there is hue vector data corresponding to the number of pixels when the training image 70 is taken.

[0069] Figure 3 An example of representing the label value 77. Regarding the label value, different label values 77 are assigned according to the type of cell and the presence or absence of the characteristics of each cell.

[0070] <Overview of Deep Learning>

[0071] Take Figure 2 as an example to illustrate the overview of neural network training. Both the first neural network 50 and the second neural network 51 are preferably convolutional neural networks. The number of nodes in the input layer 50a of the first neural network 50 corresponds to the product of the number of pixels of the input training data 75 and the number of brightness and hue contained in the image (for example, 3 in the above example: brightness 72y, first hue 72cb, and second hue 72cr). The hue vector data 74 is input as a set 76 to the input layer 50a of the first neural network 50. The label value 77 of each pixel of the training data 75 is used as the output layer 50b of the first neural network to train the first neural network 50.

[0072] The first neural network 50 extracts feature amounts for the types of the above-mentioned morphological cells and cell characteristics based on the training data 75. The output layer 50b of the first neural network outputs the results reflecting the above feature amounts. Each result output from the softmax function of the first neural network 50 is input to the input layer 51a of the second neural network 51. In addition, the cell morphologies of cells belonging to a certain cell line are similar, so the deep learning algorithm having the second neural network 51, i.e., the deep learning algorithm 51, is trained to be further specialized in identifying the types of specific cells in morphology and the characteristics of specific cells. Therefore, the label value 77 of the training data 75 is also input to the output layer of the second neural network.

[0073] Figure 2 The symbols 50c and 51c represent the intermediate layers.

[0074] The first deep learning algorithm 60 having the first neural network 60 and the second deep learning algorithm 61 having the second neural network 61 trained in this way are combined and used as an identifier for identifying which one of several cell types that the analysis target cell belongs to a certain cell group and is morphologically classified.

[0075] <Image Analysis Method>

[0076] Figure 4An example of an image analysis method is shown. In the image analysis method, analysis data 81 is generated from an analysis image 78 obtained by photographing an analysis target cell. The analysis image 78 is an image obtained by photographing an analysis target cell. For example, a well-known photographing device such as an optical microscope or a virtual slide scanner can be used to obtain the analysis image 78. Figure 4 In the example of Figure 4 , the original image photographed by Sysmex DI-60 at 360 pixels × 365 pixels is reduced to 255 pixels × 255 pixels in the same way as the training image 70 for the analysis image 78, but it is not necessary to reduce it. The number of pixels of the analysis image 78 only needs to be able to perform analysis and is not particularly limited, and it is preferably more than 100 pixels on one side. In addition, Figure 4 In the example of Figure 4 , there are red blood cells around the segmented neutrophil, and the image can also be cropped to retain only the target cell. At least one cell of the analysis target cell (red blood cells and normal-sized platelets can be included) should be contained in one image, and the pixels belonging to the analysis target cell should account for about 1 / 9 or more of the total pixels of the image to be used as the analysis image 78.

[0077] As an example, a preferred technical solution is: In this embodiment, the photographing device is photographed using RGB colors and CMY colors, etc. Preferably, the color image represents the shades or brightness of each primary color such as red, green, and blue or cyan, magenta, and yellow with 24-bit values (8 bits × 3 colors). The analysis image 78 only needs to contain at least one hue and the shade or brightness of that hue, and it is more preferably to contain at least two hues and the shade or brightness of each hue. The information containing the hue and the shade or brightness of that hue is also called the color tone.

[0078] For example, convert from RGB color to a format containing brightness information and hue information. Formats containing brightness information and hue information can include YUV (such as YCbCr, YPbPr, YIQ, etc.). Here, the conversion to the YCbCr format is taken as an example for explanation. Here, since the analysis image is in RGB color, it is converted to brightness 79Y, the first hue (such as the blue system) 79Cb, and the second hue (such as the red system) 79Cr. The conversion from RGB to YCbCr can be performed by a well-known method. For example, it can be converted from RGB to YCbCr in accordance with the international standard ITU-R BT.601. As Figure 4 shown, the converted brightness 79Y, the first hue 79Cb, and the second hue 79Cr can be respectively represented by the rows and columns of hierarchical values (hereinafter also called the color tone rows and columns 79y, 79cb, 79cr). The brightness 79Y, the first hue 79Cb, and the second hue 79Cr are each represented by 256 levels from 0 to 255 levels. Here, the analysis image can also be converted using the three primary colors of red R, green G, and blue B, or the three primary colors of cyan C, magenta M, and yellow instead of the brightness 79Y, the first hue 79Cb, and the second hue 79Cr.

[0079] Next, based on the hue rows and columns 79y, 79cb, and 79cr, hue vector data 80 is generated for each pixel by combining the three level values of the luminance 79y, the first hue 79cb, and the second hue 79cr. A set of the hue vector data 80 generated from one analysis target image 78 is used as analysis data 81.

[0080] Preferably, the generation of the analysis data 81 and the generation of the training data 75 are performed under the same conditions at least for the shooting conditions and the generation conditions of the vector data input to the neural network from each image.

[0081] The analysis data 81 is input to the input layer 60a of the first neural network 60 that constitutes the first deep learning algorithm 60 that has been trained. The first deep learning algorithm extracts feature amounts from the analysis data 81, and outputs the result from the output layer 60c of the first neural network 60. The value output from the output layer 60c is the probability of each item of the classification and features of the analysis target cell contained in the analysis target image belonging to the morphological cells input as the training data.

[0082] Next, the result output from the output layer 60c is input to the input layer 61a of the second neural network 61 that constitutes the second deep learning algorithm 61 that has been trained. The second deep learning algorithm 61 outputs, from the output layer 61b, the probability of each item of the classification and features of the analysis target cell contained in the analysis target image belonging to the morphological cells input as the training data, based on the input feature amounts. Further, in this probability, it is determined whether the analysis target cell contained in the analysis target image belongs to the morphological classification with the largest value, and a label value associated with the type or features of the morphological cells is output. The label value itself is output as the related data 83 of the cell morphology, or data obtained by replacing the label value with information (such as terms) indicating the presence or absence of the type or features of the morphological cells is output as the related data 83 of the cell morphology. Figure 4 In [Example 1], the recognizer outputs the label value "1" as the most likely label value 82 based on the analysis data 81, and outputs the literal data "segmented neutrophil" corresponding to this label value as the related data 83 of the cell morphology.

[0083] Figure 4 The symbols 60c and 61c represent intermediate layers.

[0084] [2. Image Analysis System 1]

[0085] <Structure of Image Analysis System 1>

[0086] The second embodiment of the present invention relates to an image analysis system.

[0087] Refer to Figure 5, the image analysis system according to the second embodiment includes a deep learning device 100A and an image analysis device 200A. The manufacturer-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A uses training data to train the neural network 50 and provides the deep learning algorithm 60 trained with the training data to the user. The deep learning algorithm constituted by the learned neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via a recording medium 98 or a network 99. The image analysis device 200A uses the deep learning algorithm constituted by the learned neural network 60 to analyze the object image.

[0088] The deep learning device 100A is constituted by, for example, a general-purpose computer and performs deep learning processing based on the flowchart described later. The image analysis device 200A is constituted by, for example, a general-purpose computer and performs image analysis processing based on the flowchart described later. The recording medium 98 is, for example, a computer-readable and non-transitory tangible recording medium such as a DVD-ROM or a USB memory.

[0089] The deep learning device 100A is connected to a photographing device 300. The photographing device 300 includes an image sensor 301 and a fluorescence microscope 302, and photographs a bright-field image of a learning specimen 308 placed on a stage 309. The training specimen 308 is stained as described above. The deep learning device 100A acquires the training image 70 photographed by the photographing device 300.

[0090] The image analysis device 200A is connected to a photographing device 400. The photographing device 400 includes an image sensor 401 and a fluorescence microscope 402, and photographs a bright-field image of an object specimen 408 placed on a stage 409. The object specimen 408 has been pre-stained as described above. The image analysis device 200A acquires the object image 78 photographed by the photographing device 400.

[0091] The photographing devices 300 and 400 can use a well-known optical microscope or virtual slide scanner having a function of photographing a specimen.

[0092] <Hardware Structure of Deep Learning Device>

[0093] Refer to Figure 6 , the manufacturer-side device 100 (deep learning device 100A, deep learning device 100B) includes a processing unit 10 (10A, 10B), an input unit 16, and an output unit 17.

[0094] The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing described later, a memory 12 that serves as a working area for data processing, a recording unit 13 that records programs and processing data described later, a bus 14 that transfers data between each unit, an interface unit 15 that performs input / output of data with external devices, and a GPU (Graphics Processing Unit) 19. The input unit 16 and the output unit 17 are connected to the processing unit 10. As an example, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the computational processing (e.g., parallel computational processing) performed by the CPU 11. That is, in the following description, the processing performed by the CPU 11 also includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

[0095] In addition, the processing unit 10 records the program related to the present invention and the neural network 50 before training in the recording unit 13 in a format such as an executable file format in advance in order to perform the processing of each step described below. Figure 8 The executable file format is, for example, a format generated by conversion from a programming language through a compiler. The processing unit 10 uses the program recorded in the recording unit 13 to perform the training processing of the first neural network 50 and the second neural network 51 before training.

[0096] In the following description, unless otherwise specified, the processing performed by the processing unit 10 refers to the processing performed by the CPU 11 based on the program and the neural network 50 stored in the recording unit 13 or the memory 12. The CPU 11 temporarily stores necessary data (intermediate data during processing, etc.) in the memory 12 as a working area, and appropriately records data such as calculation results that need to be stored long-term in the recording unit 13.

[0097] <Hardware Structure of Image Analysis Device>

[0098] Referring to Figure 7 , the user-side device 200 (image analysis device 200A, image analysis device 200B, image analysis device 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and an output unit 27.

[0099] The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing described below, a memory 22 for a work area used for data processing, a recording unit 23 that records a program and processing data described below, a bus 24 that transfers data between each unit, an interface unit 25 that performs input / output of data with an external device, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. As an example, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the computational processing (e.g., parallel computational processing) performed by the CPU 21. That is, the processing performed by the CPU 21 in the following description also includes the processing performed by the CPU 21 using the GPU 29 as an accelerator.

[0100] In addition, in order to perform the processing of each step described in the following image analysis processing, the processing unit 20 pre-records the program related to the present invention and the deep learning algorithm 60 of the trained neural network structure in the recording unit 23 in an executable file format, for example. The executable file format is, for example, a format generated by converting from a programming language through a compiler. The processing unit 20 performs processing using the program recorded in the recording unit 23, the first deep learning algorithm 60, and the second deep learning algorithm 61.

[0101] In the following description, unless otherwise specified, the processing performed by the processing unit 20 refers to the processing actually performed by the CPU 21 of the processing unit 20 based on the program and the deep learning algorithm 60 stored in the recording unit 23 or the memory 22. The CPU 21 temporarily stores necessary data (intermediate data during processing, etc.) in the memory 22 as a work area, and appropriately records data such as calculation results that need to be stored long-term in the recording unit 23.

[0102] <Function Modules and Processing Steps>

[0103] (Deep Learning Processing)

[0104] Refer to Figure 8 , the processing unit 10A of the deep learning device 100A according to the present embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. A program for causing a computer to perform deep learning processing is installed in the recording unit 13 or the memory 12 of the processing unit 10A, and these function modules are implemented by the CPU 11 executing this program. A training data database (DB) 104 and an algorithm database (DB) 105 are recorded in the recording unit 13 or the memory 12 of the processing unit 10A.

[0105] The photographing device 300 pre - photographs training images 70 in advance and stores them in advance in the recording unit 13 or the memory 12 of the processing unit 10A. The first deep - learning algorithm 50 and the second deep - learning algorithm 51 correspond to, for example, the type of morphological cells to which the cells to be analyzed belong and the characteristics of the cells, and are pre - stored in the algorithm database 105.

[0106] The processing unit 10A of the deep - learning device 100A performs Figure 9 the processing shown. If it is explained using Figure 8 each functional module shown, the training data generation unit 101 performs the processing of steps S11, S12, S16, and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15.

[0107] Using Figure 9 an example of the deep - learning processing performed by the processing unit 10A is explained.

[0108] First, the processing unit 10A acquires the training image 70. The training image 70 is obtained from the photographing device 300 through the operation of the operator, or obtained from the recording medium 98, or acquired via the network through the I / F unit 15. When acquiring the training image 70, information regarding which of the type of cells classified morphologically and / or the characteristics of the morphological cells the training image 70 represents is also acquired. The information regarding which of the type of cells classified morphologically and / or the characteristics of the morphological cells the training image 70 represents can be associated with the training image 70, or can also be input by the operator from the input unit 16.

[0109] In step S11, the processing unit 10A converts the acquired training image 70 into luminance Y, the first color phase Cb, and the second color phase Cr, and generates hue vector data 74 according to the steps described in the above - mentioned training data generation method.

[0110] In step S12, the processing unit 10A gives a label value corresponding to the hue vector data 74 based on the following information: the information associated with the training image 70, which represents which of the type of cells classified morphologically and / or the characteristics of the cells in the morphological classification; and the label value associated with the type of cells classified morphologically or the characteristics of the cells in the morphological classification stored in the memory 12 or the recording unit 13. In this way, the processing unit 10A generates the training data 75.

[0111] Figure 9 In the step S13 shown, the processing unit 10A trains the first neural network 50 and the second neural network 51 using the training data 75. After training each time using several pieces of training data 75, the training results of the first neural network 50 and the second neural network 51 are accumulated.

[0112] In the image analysis method according to this embodiment, a convolutional neural network is used. Since the stochastic gradient descent method is used, in step S14, the processing unit 10A determines whether the training results for a predetermined number of trials have been accumulated. When the training results for a certain number of trials have been accumulated (Yes), the processing unit 10A proceeds to the processing in step S15. When the training results corresponding to a certain number of trials have not been accumulated (No), the processing unit 10A proceeds to the processing in step S16.

[0113] Next, when the training results for a certain number of trials have been accumulated, in step S15, the processing unit 10A updates the connection weights w of the first neural network 50 and the second neural network 51 using the training results accumulated in step S13. In the image analysis method according to this embodiment, since the stochastic gradient descent method is used, the connection weights w of the first neural network 50 and the second neural network 51 are updated after the learning results for a certain number of trials have been accumulated. Specifically, the process of updating the connection weights w is a process of performing the gradient descent method operations shown in the following (Equation 11) and (Equation 12).

[0114] In step S16, the processing unit 10A determines whether the first neural network 50 and the second neural network 51 have been trained using a specified number of training data 75. When the training has been performed using the specified number of training data 75 (Yes), the deep learning process ends.

[0115] When the first neural network 50 and the second neural network 51 have not been trained using the specified number of training data 75 (No), the processing unit 10A proceeds from step S16 to step S17, and performs the processing from step S11 to step S16 for the next training image 70.

[0116] According to the above-described processing, the first neural network 50 and the second neural network 51 are trained to obtain the first deep learning algorithm 60 and the second deep learning algorithm 61.

[0117] (Structure of the neural network)

[0118] As described above, a convolutional neural network is used in this embodiment. Figure 10 (a) is an example of the structure of the first neural network 50 and the second neural network 51. The first neural network 50 and the second neural network 51 include input layers 50a, 51a, output layers 50b, 51b, and intermediate layers 50c, 51c between the input layers 50a, 51a and the output layers 50b, 51b. The intermediate layers 50c, 51c are composed of several layers. The number of layers constituting the intermediate layers 50c, 51c can be, for example, 5 or more.

[0119] In the first neural network 50 and the second neural network 51, several nodes 89 arranged in layers are connected between the layers. Thus, information propagates only in one direction shown by the arrow D in the figure from the input-side layers 50a and 51a to the output-side layers 50b and 51b.

[0120] (Calculation of each node)

[0121] Figure 10 (b) is a schematic diagram of the calculation of each node. Each node 89 receives several inputs and calculates one output (z). As Figure 10 (b) shows, node 89 receives 4 inputs. The total input (u) received by node 89 is represented by the following (Equation 1).

[0122]

Equation 1

[0123] u = w 1 x 1 + w 2 x 2 + w 3 x 3 + w 4 x 4 + b (Equation 1)

[0124] Each input is multiplied by a different weight value. In (Equation 1), b is a value called bias. The output (z) of the node is the output of a certain function f of the total input (u) represented by (Equation 1), and is represented by the following (Equation 2). The function f is called an activation function.

[0125]

Equation 2

[0126] z = f(u) (Equation 2)

[0127] Figure 10 (c) is a schematic diagram of the calculation between nodes. In the neural network 50, the nodes that output the result (z) represented by (Equation 1) for the total input (u) are arranged in layers. The output of the nodes in the previous layer is the input of the nodes in the next layer. In Figure 10 (c), in the example, the output of the node 89a in the left layer in the figure is the input of the node 89b in the right layer in the figure. Each node 89b in the right layer receives the output from the node 89a in the left layer. Each connection between each node 89a in the left layer and each node 89b in the right layer is multiplied by a different weight value. If the outputs of several nodes 89a in the left layer are x 1 ~x 4 respectively, then the inputs for the 3 nodes 89b in the right layer are represented by the following (Equation 3-1) to (Equation 3-3).

[0128]

Equation 3

[0129] u1 = w 11 x 1 + w 12 x 2 + w 13 x 3 + w 14 x 4 + b 1 (Equation 3-1)

[0130] u 2 = w 21 x 1 + w 22 x 2 + w 23 x 3 + w 24 x 4 + b 2 (Equation 3-2)

[0131] u 3 = w 31 x 1 + w 32 x 2 + w 33 x 3 + w 34 x 4 + b 3 (Equation 3-3)

[0132] After generalizing the above (Equation 3-1) to (Equation 3-3), we get (Equation 3-4). Here, i = 1,..., I, j = 1,..., J.

[0133]

Number 4

[0134]

[0135] Applying (Equation 3-4) to the activation function gives the output. The output is represented by the following (Equation 4).

[0136]

Number 5

[0137] z j = f(u j ) (j = 1, 2, 3) (Equation 4)

[0138] (Activation function)

[0139] In the image analysis method according to the embodiment, the rectified linear unit function is used as the activation function. The rectified linear unit function is represented by the following (Equation 5).

[0140]

Number 6

[0141] f(u) = max(u, 0) (Equation 5)

[0142] (Equation 5) is a function in which the part where u < 0 becomes u = 0 among the linear functions of z = u. In Figure 10 the example of (c), according to (Equation 5), the output of the node with j = 1 is expressed by the following equation.

[0143]

Equation 7

[0144] z 1 = max((w 11 x 1 + w 12 x 2 + w 13 x 3 + w 14 x 4 + b 1 ), 0)

[0145] (Learning of Neural Network)

[0146] Let the function represented by the neural network be y(x: w). Then, if the parameters w of the neural network change, the function y(x: w) also changes. Adjusting the function y(x: w) to make the neural network select more appropriate parameters w for the input x is called the learning of the neural network. Given an array of sets of inputs and outputs of the function represented by the neural network. When the preferred output for a certain input x is d, the given set of input and output is {(x 1 , d 1 ), (x 2 , d 2 ), …, (x n , d n )}. The set of each group represented by (x, d) is called training data. Specifically, as Figure 2 shown, the set of groups of the color density values of each pixel in the monochrome images of each color of R, G, and B and the labels of the true value images is Figure 2 the training data shown. The learning of the neural network means adjusting the weight w so that for all sets of input and output (x n , d n ), when the input x n is given, the output y(x n : w) of the neural network is as close as possible to the output d n . The error function is a measure of the degree of closeness between the function represented by the neural network and the training data.

[0147]

Equation 8

[0148] y(x n : w) ≈ dn

[0149] The error function is also called the loss function. The error function E(w) used in the image analysis method according to the embodiment is represented by the following (Equation 6). (Equation 6) is called the cross entropy.

[0150]

Number 9

[0151]

[0152] The calculation method of the cross entropy of (Equation 6) will be described. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, that is, in the final layer of the neural network, an activation function that classifies the input x into a finite number of levels according to the content is used. The activation function is called the softmax function and is represented by the following (Equation 7). In addition, the output layer 50b is arranged with the same number of nodes as the number of levels k. The total input u of each node k (k = 1,..., K) of the output layer L is given from the output of the previous layer L - 1 by u k (L) Given this, the output of the k-th node of the output layer is represented by the following (Equation 7).

[0153]

Number 10

[0154]

[0155] (Equation 7) is the softmax function. The sum of the outputs y 1 , …, y K determined by (Equation 7) is always 1.

[0156] If each level is represented as C 1 , …, C K , then the output y K (i.e., z k (L) ) of the k-th node of the output layer L represents the probability that the given input x belongs to the level C K . Refer to the following (Equation 8). The input x is classified into the level with the highest probability represented by (Equation 8).

[0157]

Number 11

[0158]

[0159] In the learning of a neural network, the function represented by the neural network is regarded as a model of posterior probabilities at various levels. Under such a probability model, the likelihood of the weights w with respect to the training data is evaluated, and the weights w that maximize the likelihood are selected.

[0160] The target output d obtained by the softmax function of (Equation 7) will only pass when the correct level is output. n Is denoted as 1, and the other outputs are denoted as 0. When the target output is represented by d n =[d n1 , …, d nK In this vector form, for example, when the input x n The correct level is C 3 Then only the target output d n3 Is denoted as 1, and the other target outputs are denoted as 0. Encoded in this way, the posterior distribution is represented by the following (Equation 9).

[0161]

Equation 12

[0162]

[0163] The likelihood L(w) of the weights w with respect to the training data {(x n , d n )} (n = 1, …, N) is represented by the following (Equation 10). Taking the logarithm of the likelihood L(w) and reversing the sign leads to the error function of (Equation 6).

[0164]

Equation 13

[0165]

[0166] Learning means minimizing the error function E(w) calculated based on the training data with respect to the parameters w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is represented by (Equation 6).

[0167] Minimizing the error function E(w) with respect to the parameter w has the same meaning as finding the local minimum point of the function E(w). The parameter w is the weight of the connection between nodes. The minimum point of the weight w is obtained by iterative calculation, and this iterative calculation starts from an arbitrary initial value and repeatedly updates the parameter w. An example of the above operation is the gradient descent method.

[0168] In the gradient descent method, the vector represented by the following (Equation 11) is used.

[0169]

Equation 14

[0170]

[0171] In the gradient descent method, the process of repeatedly moving the current value of the parameter w in the direction of the negative gradient (i.e., ) is performed multiple times. If the current weight is w (t) , and the moved weight is w (t+1) , the calculation performed by the gradient descent method is represented by the following (Equation 12). The value t indicates the number of times the parameter w is moved.

[0172] [Equation 15]

[0173]

[0174] Label [Equation 16]

[0175] ∈

[0176] is a constant that determines the magnitude of the update amount of the parameter w and is called the learning coefficient. By repeatedly performing the calculation represented by (Equation 12), the error function E(w (t) ) decreases as the value t increases, and the parameter w reaches the minimum point.

[0177] In addition, the calculation of (Equation 12) can be performed for all training data (n = 1,..., N), or the calculation of (Equation 12) can be performed only for a part of the training data. The gradient descent method performed only for a part of the training data is called the stochastic gradient descent method. The stochastic gradient descent method is used in the image analysis method related to the implementation manner.

[0178] (Image Analysis Processing)

[0179] Figure 11 FIG. shows a functional block diagram of an image analysis device 200A that performs image analysis processing, up to generating relevant data 83 of cell morphology from the analysis target image 78. The processing unit 20A of the image analysis device 200A includes an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. The program of the present invention for causing a computer to execute image analysis processing is installed in the recording unit 23 or the memory 22 of the processing unit 20A, and these functional modules are implemented by the CPU 21 executing the program. The training data database (DB) 104 and the algorithm database (DB) 105 are provided from the deep learning device 100A through the recording medium 98 or the network 99 and recorded in the recording unit 23 or the memory 22 of the processing unit 20A.

[0180] The imaging device 400 captures the analysis target image 78 and stores it in the recording unit 23 of the processing unit 20A or the memory 22. The first deep learning algorithm 60 and the second deep learning algorithm 61 that have completed training and contain connection weights w exist in the algorithm database 105 corresponding to, for example, the type of cells classified based on morphology to which the analysis target cells belong and the characteristics of the cells, and function as part of a program that causes a computer to execute image analysis processing. That is, the first deep learning algorithm 60 and the second deep learning algorithm 61 are used in a computer having a CPU and a memory, and are used for: identifying to which one of several cell types classified morphologically and belonging to a certain cell group the analysis target cells belong, and generating relevant data 83 on the cell morphology. In addition, the generated data is output as needed. In addition, the CPU 21 of the processing unit 20A causes the computer to perform calculations or processing of unique information corresponding to the usage purpose. Specifically, the CPU 21 of the processing unit 20A uses the first deep learning algorithm 60 and the second deep learning algorithm 61 recorded in the recording unit 23 or the memory 22 to generate relevant data 83 on the cell morphology. The CPU 21 of the processing unit 20A inputs the analysis data 81 to the input layer 60a, and outputs the feature amount of the analysis target image 78 calculated by the first deep learning algorithm 60 from the output layer 60b. The CPU 21 of the processing unit 20A inputs the feature amount output from the first deep learning algorithm 60 to the input layer 61a of the second deep learning algorithm, and outputs from the output layer 61b the label value of the type of cells or the characteristics of the cells classified morphologically to which the cells contained in the analysis target image belong and are identified.

[0181] Use Figure 11 If the respective functional modules shown are used for explanation, the analysis data generation unit 201 performs the processes of steps S21 and S22. The analysis data input unit 202 performs the processes of steps S23, S24, S25, and S27. The analysis unit 203 performs the process of step S26.

[0182] Use Figure 12 An example of the image analysis processing performed by the processing unit 20A will be described up to the generation of relevant data 83 on the cell morphology from the analysis target image 78.

[0183] First, the processing unit 20A acquires the analysis target image 78. The analysis target image 78 is acquired from the imaging device 400 through the operation of the user, or from the recording medium 98, or via the network through the I / F unit 25.

[0184] In step S21, similar to Figure 9 step S11 shown, the acquired analysis target image 78 is converted into luminance Y, first color phase Cb, and second color phase Cr, and hue vector data 80 is generated according to the steps described in the above method for generating analysis data.

[0185] Next, in step S22, the processing unit 20A generates the analysis data 81 from the hue vector data 80 according to the steps described in the generation method of the above analysis data.

[0186] Next, in step S23, the processing unit 20A acquires the first deep learning algorithm and the second deep learning algorithm existing in the algorithm database 105.

[0187] Next, in step S24, the processing unit 20A inputs the analysis data 81 into the first deep learning algorithm. The processing unit 20A inputs the feature amount output from the first deep learning algorithm into the second deep learning algorithm according to the steps described in the above image analysis method, and outputs the label value to which the analysis object cell contained in the analysis image belongs judged from the second deep learning algorithm. The processing unit 20A stores this label value in the memory 22 or the recording unit 23.

[0188] In step S25, the processing unit 20A determines whether all of the first acquired analysis images 78 have been recognized. If the recognition of all the analysis images 78 has been completed (yes), the process proceeds to step S26, and the analysis result including the related data 83 on the cell morphology is output. If the recognition of all the analysis images 78 has not been completed (no), the process proceeds to step S27, and the processing from step S21 to step S25 is performed on the unrecognized analysis images 78.

[0189] Through this embodiment, regardless of the skills of the examiner, the types of cells and the characteristics of the cells based on morphological classification can be recognized, and the inconsistency in morphological examination can be suppressed.

[0190] <Computer Program>

[0191] The present invention includes a computer program for image analysis that causes a computer to execute the processes of steps S11 to S17 and / or S21 to S27 for analyzing cell morphology.

[0192] In addition, an embodiment of the present invention relates to a program product such as a storage medium storing the computer program. That is, the computer program is stored in a semiconductor storage element such as a hard disk or a flash memory, or a storage medium such as an optical disc. Regarding the storage format of the program stored in the storage medium, there is no particular limitation as long as the prompting device can read the program. It is preferably non-volatile storage in the storage medium.

[0193] [3. Image Analysis System 2]

[0194] <Structure of Image Analysis System 2>

[0195] Other technical solutions of the image analysis system will be described.

[0196] Figure 13 This represents a structural example of the second image analysis system. The second image analysis system includes a user-side device 200, and the user-side device 200 operates as an integrated image analysis device 200B. The image analysis device 200B is composed of, for example, a general-purpose computer and performs two types of processing: the deep learning processing and the image analysis processing described in the above image analysis system 1. That is, the second image analysis system is an independent operation type system that performs deep learning and image analysis on the user side. The integrated image analysis device 200B provided on the user side of the second image analysis system undertakes the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment.

[0197] Figure 13 In this case, the image analysis device 200B is connected to the imaging device 400. The imaging device 400 captures training images 70 during deep learning processing and captures analysis target images 78 during image analysis processing.

[0198] <Hardware Structure>

[0199] The hardware structure of the image analysis device 200B is the same as Figure 7 the hardware structure of the user-side device 200 shown.

[0200] <Functional Modules and Processing Steps>

[0201] Figure 14A functional block diagram of the image analysis device 200B is shown. The processing unit 20B of the image analysis device 200B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. A program for causing a computer to execute deep learning processing and image analysis processing is installed in the recording unit 23 or the memory 22 of the processing unit 20B, and these functional blocks are implemented by the CPU 21 executing this program. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 23 or the memory 22 of the processing unit 20B, and are common in both deep learning and image analysis processing. The first neural network 60 and the second neural network 61 that have completed training pre-exist in the algorithm database 105 corresponding to, for example, the type of cell based on morphological classification to which the analysis target cell belongs and the characteristics of the cell. The connection weights w are updated through deep learning processing and exist as the deep learning algorithm 60 in the algorithm database 105. In addition, the training image 70 is pre-captured by the imaging device 400 and pre-stored in the training data database (DB) 104, the recording unit 23 of the processing unit 20B, or the memory 22. The analysis target image 78 of the analysis target specimen is pre-captured by the imaging device 400 and pre-stored in the recording unit 23 of the processing unit 20B or the memory 22.

[0202] The processing unit 20B of the image analysis device 200B performs Figure 9 the processing shown during deep learning processing and Figure 12 the processing shown during image analysis processing. Using Figure 13 each functional block shown for explanation, during deep learning processing, the training data generation unit 101 performs the processing of steps S11, S12, S16, and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15. During image analysis processing, the analysis data generation unit 201 performs the processing of steps S21 and S22. The analysis data input unit 202 performs the processing of steps S23, S24, S25, and S27. The analysis unit 203 performs the processing of step S26.

[0203] The steps of the deep learning processing and the steps of the image analysis processing performed by the image analysis device 200B are the same as the steps respectively performed by the deep learning device 100A and the image analysis device 200A. Only the image analysis device 200B obtains the training image 70 from the imaging device 400.

[0204] In the image analysis device 200B, the user can confirm the recognition accuracy of the recognizer. In case the recognition result of the recognizer is different from the recognition result obtained by the user's image observation, the analysis data 81 can be used as the training data 75, and the recognition result obtained by the user's image observation can be used as the label value 77 to retrain the first deep learning algorithm and the second deep learning algorithm. In this way, the training efficiency of the first neural network 50 and the second neural network 51 can be further improved.

[0205] [3. Image Analysis System 3]

[0206] <Structure of Image Analysis System 3>

[0207] Other technical solutions of the image analysis system will be described.

[0208] Figure 15 Fig. shows a structural example of the third image analysis system. The third image analysis system includes a manufacturer-side device 100 and a user-side device 200. The manufacturer-side device 100 operates as an integrated image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is composed of, for example, a general-purpose computer and is a device on the cloud server side that performs both the deep learning process and the image analysis process described in the above image analysis system 1. The terminal device 200C is composed of, for example, a general-purpose computer and is a user-side terminal device that sends the analysis target image to the image analysis device 100B via the network 99 and receives the analysis result from the image analysis device 100B via the network 99.

[0209] The integrated image analysis device 100B provided on the manufacturer side of the third image analysis system undertakes the functions of both the deep learning device 100A and the image analysis device 200A. On the other hand, the third image analysis system includes a terminal device 200C and provides the input interface for the analysis target image 78 and the output interface for the analysis result to the user-side terminal device 200C. That is, the third image analysis system is a cloud service type system in which the manufacturer side that performs the deep learning process and the image analysis process provides the input interface for the analysis target image 78 to the user side and provides the relevant data 83 of the cell morphology to the user side. The input interface and the output interface can be integrated.

[0210] The image analysis device 100B is connected to the imaging device 300 and acquires the training image 70 captured by the imaging device 300.

[0211] The terminal device 200C is connected to the imaging device 400 and acquires the analysis target image 78 captured by the imaging device 400.

[0212] <Hardware Structure>

[0213] The hardware structure of the image analysis device 100B is the same as that of Figure 6 the hardware structure of the manufacturer-side device 100 shown. The hardware structure of the terminal device 200C is the same as that of Figure 7 the hardware structure of the user-side device 200 shown.

[0214] <Function Modules and Processing Steps>

[0215] Figure 16 It shows a functional module diagram of the image analysis device 100B. The processing unit 10B of the image analysis device 100B includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, and an analysis unit 203. A program for causing a computer to execute deep learning processing and image analysis processing is installed in the recording unit 13 or the memory 12 of the processing unit 10B, and these functional modules are implemented by the CPU 11 executing this program. The training data database (DB) 104 and the algorithm database (DB) 105 are stored in the recording unit 13 or the memory 12 of the processing unit 10B, and are common in both deep learning and image analysis processing. The first neural network 50 and the second neural network 51 are pre-existing in the algorithm database 105 corresponding to, for example, the type of cells classified based on morphology to which the cells to be analyzed belong and the characteristics of the cells. The connection weights w are updated through deep learning processing and exist as the first deep learning algorithm 60 and the second deep learning algorithm 61 in the algorithm database 105.

[0216] The training image 70 is pre-shot by the imaging device 300 and pre-recorded in the training data database (DB) 104 or the recording unit 13 or the memory 12 of the processing unit 10B. The image 78 to be analyzed is shot by the imaging device 400 and pre-recorded in the recording unit 23 or the memory 22 of the processing unit 20C of the terminal device 200C.

[0217] The processing unit 10B of the image analysis device 100B performs the Figure 9 processing shown during deep learning processing and performs the Figure 12 processing shown during image analysis processing. Using Figure 16 each functional module shown for explanation, during deep learning processing, the training data generation unit 101 performs the processing of steps S11, S12, S16, and S17. The training data input unit 102 performs the processing of step S13. The algorithm update unit 103 performs the processing of steps S14 and S15. During image analysis processing, the analysis data generation unit 201 performs the processing of steps S21 and S22. The analysis data input unit 202 performs the processing of steps S23, S24, S25, and S27. The analysis unit 203 performs the processing of step S26.

[0218] The steps of the deep learning process and the image analysis process performed by the image analysis device 100B are the same as the steps respectively performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment.

[0219] The processing unit 10B receives the image 78 to be analyzed from the terminal device 200C on the user side, and generates the training data 75 according to the steps S11 to S17 shown in Figure 9 the figure.

[0220] Figure 12 In step S26 shown in the figure, the processing unit 10B sends the analysis result including the relevant data 83 of the cell morphology to the terminal device 200C on the user side. In the terminal device 200C on the user side, the processing unit 20C outputs the received analysis result to the output unit 27.

[0221] As described above, the user of the terminal device 200C can obtain the relevant data 83 of the cell morphology as the analysis result by sending the image 78 to be analyzed to the image analysis device 100B.

[0222] According to the image analysis device 100B according to the third embodiment, the user can use the recognizer without obtaining the training data database 104 and the algorithm database 105 from the deep learning device 100A. Thus, the service of recognizing the types of cells classified based on morphology and the characteristics of cells can be provided as a cloud service.

[0223] [4. Other Embodiments]

[0224] The present invention has been described with a general outline and specific embodiments, but the present invention is not limited to the above general outline and each embodiment.

[0225] In the present invention, a method of converting hue into brightness Y, first hue Cb, and second hue Cr to generate the training data 75 is exemplified, but the conversion of hue is not limited thereto. It is also possible not to convert the hue, for example, directly use the three primary colors of red (R), green (G), and blue (B). Or it can be two primary colors obtained by subtracting any one hue from the primary colors. Or it can be only any one (for example, green (G)) of the three primary colors of red (R), green (G), and blue (B), that is, one primary color. It is also possible to convert to the three primary colors of cyan (C), magenta (M), and yellow (Y). For example, the image 78 to be analyzed is not limited to a color image of the three primary colors of red (R), green (G), and blue (B), and can be a color image of two primary colors, as long as it is an image containing one or more primary colors.

[0226] In the above-described training data generation method and analysis data generation method, in step S11, the processing units 10A, 20B, and 10B generate the hue rows 72y, 72cb, and 72cr from the training image 70. However, the training image 70 may also be an image that has been converted to luminance Y, first color phase Cb, and second color phase Cr. That is, the processing units 10A, 20B, and 10B may directly obtain luminance Y, first color phase Cb, and second color phase Cr from, for example, a virtual slide scanner at the beginning. Similarly, in step S21, the processing units 20A, 20B, and 10B generate the hue rows 79y, 79cb, and 79cr from the analysis target image 78. However, the processing units 20A, 20B, and 10B may directly obtain luminance Y, first color phase Cb, and second color phase Cr from, for example, a virtual slide scanner at the beginning.

[0227] In addition to RGB and CMY, YUV and CIE L ＊ a ＊ b ＊ and other color spaces can also be used for image acquisition and hue conversion.

[0228] In the hue vector data 74 and the hue vector data 80, the hue information is stored in the order of luminance Y, first color phase Cb, and second color phase Cr for each pixel. However, the storage order and processing order of the hue information are not limited to this. However, it is preferable that the arrangement order of the hue information in the hue vector data 74 is the same as the arrangement order of the hue information in the hue vector data 80.

[0229] In each image analysis system, the processing units 10A and 10B are integrated into one device. However, the processing units 10A and 10B do not necessarily have to be designed as an integrated device. They can be designed such that the CPU 11, memory 12, recording unit 13, GPU 19, etc. are arranged elsewhere and connected by a network. The processing units 10A and 10B, input unit 16, and output unit 17 do not necessarily have to be arranged in one place either. They can be designed to be arranged in different places, connected to each other by a network, and capable of communicating. The processing units 20A, 20B, and 20C are the same as the processing units 10A and 10B.

[0230] In the above-described first to third embodiments, the respective functional modules of the training data generation unit 101, training data input unit 102, algorithm update unit 103, analysis data generation unit 201, analysis data input unit 202, and analysis unit 203 are executed in a single CPU 11 or a single CPU 21. However, these respective functional modules do not necessarily have to be executed in a single CPU. They can also be executed in a distributed manner among several CPUs. In addition, these functional modules can be executed in a distributed manner among several GPUs, or can be executed in a distributed manner among several CPUs and several GPUs.

[0231] In the above-described second and third embodiments, for performing Figure 9 andFigure 12 The programs for processing the respective steps of the description are pre-recorded in the recording units 13 and 23. This solution can also be replaced by the following solution: installing the program in the processing units 10B and 20B from a computer-readable and non-transitory tangible recording medium 98 such as a DVD-ROM or a USB memory. Alternatively, it can be designed such that the processing units 10B and 20B are connected to the network 99, and the program is downloaded and installed from an external server (not shown) via the network 99, for example.

[0232] In each image analysis system, the input units 16 and 26 are input devices such as a keyboard or a mouse, and the output units 17 and 27 are display devices such as a liquid crystal display. However, alternative technical solutions can also be adopted, that is, integrating the input units 16 and 26 and the output units 17 and 27 and designing them as a touch screen type display device. Or it can also be designed such that the output unit 17 and 27 are constituted by a printer or the like.

[0233] In each of the above image analysis systems, the imaging device 300 is directly connected to the deep learning device 100A or the image analysis device 100B, but the imaging device 300 can also be connected to the deep learning device 100A or the image analysis device 100B via the network 99. The same applies to the imaging device 400. The imaging device 400 is directly connected to the image analysis device 200A or the image analysis device 200B, but the imaging device 400 can also be connected to the image analysis device 200A or the image analysis device 200B via the network 99.

[0234] [5. Effect of Deep Learning Algorithm]

[0235] To verify the effect of the deep learning algorithm, the recognition accuracy of cells using an existing machine learning-based cell recognition method and a cell recognition method using the deep learning algorithm of the present invention was compared.

[0236] A peripheral blood smear specimen was prepared using the smear specimen preparation device SP-1000i, and cells were imaged using the blood imaging automatic analysis device DI-60. The staining was May-Grünwald-Giemsa staining.

[0237] Cell recognition using an existing machine learning method was performed using the blood imaging automatic analysis device DI-60. In addition, three people including a physician and a skilled technician observed the images for verification.

[0238] Figure 17 This is the comparison result of the blood cell classification accuracy. When using the deep learning algorithm, discrimination with higher accuracy can be performed than the existing method.

[0239] Next, it was investigated whether the deep learning algorithm of the present invention could recognize the morphological features appearing in myelodysplastic syndrome (MDS). The results are as Figure 18 shown.

[0240] As Figure 18 shown, nuclear morphological abnormalities, vacuolization, abnormal particle distribution, etc. can all be identified with high precision.

[0241] As described above, the deep learning algorithm of the present invention can identify the types of cells classified based on morphology and the characteristics of cells with high precision.

[0242] Symbol Explanation

[0243] 200 Image analysis device

[0244] 10 Processing unit

[0245] 50, 51 Deep learning algorithms before training

[0246] 50a, 51a Input layer

[0247] 50b, 51b Output layer

[0248] 60, 61 Trained deep learning algorithms

[0249] 75 Training data

[0250] 81 Analysis data

[0251] 83 Data related to cell morphology

Claims

1. An image analysis method, comprising the following steps: Inputting the analysis data into a recognizer having a neural network structure, the analysis data being generated from an image of an analysis target cell and containing information related to the analysis target cell, and the analysis target cell being a cell in peripheral blood; Calculating the morphological probability of the analysis target cell by using the recognizer; and Outputting an analysis result of the analysis target cell based on the calculated morphological probability, wherein, The morphological classification corresponding to the calculated morphological probability indicates an abnormal manifestation in the blood cells, and the abnormal manifestations include: nuclear morphological abnormality in blast cells, vacuoles in blast cells, granules in blast cells, giant band neutrophils, nuclear morphological abnormality in band neutrophils, abnormal granule distribution in band neutrophils, giant segmented neutrophils, nuclear morphological abnormality in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hyperlobulated nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

2. The image analysis method according to claim 1, wherein The input and the calculation are performed for a plurality of analysis target cells, and The output includes outputting an analysis result based on the morphological probabilities of the plurality of analysis target cells calculated.

3. The image analysis method according to claim 1, wherein The output includes outputting a label associated with the morphological classification as the analysis result.

4. The image analysis method according to claim 1, further comprising Calculating the type probability of the analysis target cell by using the recognizer, wherein, The output includes outputting the analysis result based on the calculated morphological probability and the calculated type probability.

5. The image analysis method according to claim 4, wherein The type classification corresponding to the calculated type probability indicates the type of blood cells, and The types include segmented neutrophils, band neutrophils, metamyelocytes, myelocytes, blast cells, lymphocytes, monocytes, eosinophils, basophils, erythroblasts or megakaryocytes.

6. The image analysis method according to claim 1, further comprising Cropping the image of the analysis target cell, wherein The analysis data is generated according to the cropped image.

7. The image analysis method according to claim 1, further comprising Taking an image of the analysis target cell by an imaging device, and Generating the analysis data according to the image taken by the imaging device.

8. The image analysis method according to claim 1, further comprising Making a blood smear specimen in which the analysis target cell is stained, Taking an image of the stained analysis target cell by an imaging device, and Generating the analysis data according to the image taken by the imaging device.

9. The image analysis method according to claim 8, wherein The analysis target cell is stained by Wright staining, Giemsa staining, Swiss Giemsa staining or May-Grünwald Giemsa staining.

10. An image analysis device, comprising: A processing unit, including a processor and a memory, wherein the processing unit is configured to: Input the parsed data into an identifier having a neural network structure, the parsed data being generated from an image of a parsed target cell and including information related to the parsed target cell, the parsed target cell being a cell in peripheral blood; Calculate the morphological probability of the parsed target cell by using the identifier; and Output an analysis result of the parsed target cell based on the calculated morphological probability, wherein, The morphological classification corresponding to the calculated morphological probability indicates an abnormal manifestation in the blood cells, and the abnormal manifestations include: abnormal nuclear morphology in blast cells, vacuoles in blast cells, granules in blast cells, giant banded neutrophils, abnormal nuclear morphology in banded neutrophils, abnormal granule distribution in banded neutrophils, giant segmented neutrophils, abnormal nuclear morphology in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hyper-segmented nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

11. The image analysis device according to claim 10, wherein The processing unit is configured to perform the input and the calculation for a plurality of parsed target cells, and Output an analysis result based on the morphological probabilities of the plurality of parsed target cells calculated.

12. The image analysis device according to claim 10, wherein The processing unit is configured to output a label associated with the morphological classification as the analysis result.

13. The image analysis device according to claim 10, wherein The processing unit is configured to calculate the species probability of the parsed target cell by using the identifier, and Output the analysis result based on the calculated morphological probability and the calculated species probability.

14. The image analysis device according to claim 13, wherein The species classification corresponding to the calculated species probability indicates the species of the blood cells, and The species include segmented neutrophils, banded neutrophils, metamyelocytes, myelocytes, blast cells, lymphocytes, monocytes, eosinophils, basophils, erythroblasts or megakaryocytes.

15. The image analysis device according to claim 10, wherein The processing unit is configured to crop the image of the parsed target cell, and Generate the parsed data according to the cropped image.

16. An image analysis system, Comprising: An imaging device configured to capture an image of a parsed target cell, the image including information related to the parsed target cell, and the parsed target cell being a cell in peripheral blood; And A processing unit, including a processor and a memory, wherein the processing unit is configured to: Generate parsed data according to the image captured by the imaging device; Input the parsed data into an identifier having a neural network structure; Calculate the morphological probability of the parsed target cell by using the identifier; And Output the analysis result of the analyzed object cell based on the calculated morphological probability, wherein, The morphological classification corresponding to the calculated morphological probability indicates abnormal manifestations in blood cells, and the abnormal manifestations include: abnormal nuclear morphology in blast cells, vacuoles in blast cells, granules in blast cells, giant banded neutrophils, abnormal nuclear morphology in banded neutrophils, abnormal granule distribution in banded neutrophils, giant segmented neutrophils, abnormal nuclear morphology in segmented neutrophils, pseudo-Pelger in segmented neutrophils, hyper-segmented nuclei in segmented neutrophils, or abnormal granule distribution in segmented neutrophils.

17. The image analysis system according to claim 16, further comprising A smear specimen preparation device configured to prepare a blood smear specimen stained with the analyzed object cell, wherein the imaging device is configured to capture an image of the stained analyzed object cell.

18. The image analysis system according to claim 17, wherein The smear specimen preparation device is configured to prepare the blood smear specimen stained with the analyzed object cell by Wright staining, Giemsa staining, Swiss Giemsa staining or May-Grünwald Giemsa staining.

Images