Sample analysis method and image processing method

Abstract

To provide technology with which, when analyzing a pathological specimen in cell units, it is possible to cope with multiple immunostaining also, as well as specify the position of each individual cell and conduct measurement of the same even when a cell density in the specimen is high.SOLUTION: Multiple kinds of staining is applied in order to the specimen to be evaluated, and image processing is performed, for at least two sessions of staining, using a plurality of sample images in which the sample after staining is imaged. This image processing comprises: a step for extracting an internal region which is inward of a cell membrane among cells in the sample, on the basis of at least one of the sample images; a step for extending the extracted internal region to the outside and thereby determining a cell region that corresponds to each individual cell included in the sample; and a step for carrying out image cytometry on the cell region with respect to the plurality of sample images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for analyzing images of pathological specimens on a cell-by-cell basis and an image processing method to support the same, and more particularly to analysis using immunohistochemical techniques. [Background technology]

[0002] In recent years, in the field of immunohistochemistry, so-called single-cell analysis has been attracting attention, in which a specimen is not only viewed as a group of cells, but also focuses on the individual cells that compose it, and measures the expression levels and locations of biological substances such as proteins and related substances on a cell-by-cell basis. A representative analytical technique is flow cytometry, which circulates a fluid containing dispersed cells through a thin channel and measures the cells that pass through the channel one by one.

[0003] This flow cytometry technique has a problem in that the positional information of cells in a specimen is lost. For this reason, a technique has been devised that uses image processing techniques to directly measure individual cells from images of tissue slices. For example, in the technique described in Patent Document 1, different biological substances in the same specimen are stained with a dye that can be observed in a bright field and with a dye that produces fluorescence, and the positions of the biological substances to be observed are identified by comparing the captured bright field images with the fluorescent images, and the amount of expression is identified from the measurement results of the amount of fluorescence. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2015 / 045962 Summary of the Invention [Problem to be solved by the invention]

[0005] The conventional technology described in the above Patent Document 1 is capable of detecting the expression location and expression amount of a specific biological substance with high accuracy, but has the following problems to be solved. That is, Patent Document 1 does not describe in detail a method for identifying the location of a biological substance, and it may be difficult to clearly separate individual cells, especially in a specimen in which cells are densely packed. In addition, with the recent improvement of multiple immunostaining technology, the same specimen is stained with multiple types of stains to measure multiple types of biological substances, but the above conventional technology has restrictions on the staining method and cannot be immediately applied to such multiple immunostained specimens.

[0006] Specifically, tissue sections with a thickness of 4 μm or more are generally used as specimens for multiple immunostaining, but in such tissue sections, cells overlap in the thickness direction, making it difficult to separate individual cells in the image. This problem is particularly noticeable in specimens with high cell density. Making the tissue section thinner is advantageous in terms of identifying cells, but another problem is that the specimen is easily damaged, limiting the number of times it can be stained.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that can analyze pathological specimens on a cell-by-cell basis, is compatible with multiple immunostaining, and can identify and measure the position of individual cells even when the cell density in the specimen is high. [Means for solving the problem]

[0008] In order to achieve the above object, one aspect of the present invention includes a step of sequentially staining a specimen to be evaluated with a plurality of types of staining, and capturing images of the specimen after at least two of the staining processes to obtain a plurality of specimen images; and determining whether or not a cell membrane of a cell in the specimen is detected based on at least one of the specimen images. As an inner region, the cell membranea step of extracting a closed region surrounded by an internal region of the specimen, a step of determining a cellular region corresponding to each individual cell contained in the specimen by expanding the extracted internal region outward, and a step of performing image cytometry on the cellular region for a plurality of the specimen images.

[0009] Another aspect of the present invention is an image processing method using a plurality of specimen images obtained by sequentially staining a specimen to be evaluated with a plurality of types of staining and capturing images of the specimen after at least two of the staining processes, the method comprising: determining a cell membrane of a cell in the specimen based on at least one of the specimen images; As an inner region, the cell membrane the step of extracting a closed region surrounded by the inner region, the step of determining a cellular region corresponding to each individual cell contained in the specimen by expanding the extracted inner region outward, and the step of performing image cytometry on the cellular region for a plurality of the specimen images.

[0010] In these inventions, "image cytometry" refers to image processing that measures at least one piece of quantitative information, such as the shape and size of cells in a specimen, and the expression level of biological substances (e.g., proteins) in the cells, on a cell-by-cell basis based on image data of the specimen image.

[0011] In the invention configured in this way, the area inside the cell membrane of the cells appearing in the specimen image is extracted. Even if multiple cells are in contact with each other, the cells are isolated from each other in the area inside the cell membrane. Therefore, even if the cells are in contact with each other through the cell membrane, they can be identified as individual cells. In other words, at this stage, it is possible to grasp the position and number of each cell in the specimen. Then, by determining the cell area by expanding the internal area outward, the position occupied by the cell including the cell membrane in the image can be determined individually.

[0012] When many cells are densely packed together, their boundaries tend to be unclear. In particular, when trying to extract a cell region as a structure including the cell membrane, multiple cells may be extracted as one cell due to the unclear boundaries. In view of this, in the present invention, the internal region inside the cell membrane is first identified in the specimen image, and then expanded to determine the cell region corresponding to each cell. Therefore, even in tissues with high cell density, it is possible to accurately extract the individual cells that make up the tissue.

[0013] Once the areas occupied by individual cells in the specimen image have been identified, it is possible to apply image cytometry, an image processing technique that extracts quantitative information about each cell in the image, to individually analyze each cell that makes up the specimen tissue and obtain quantitative information about each one. Effect of the Invention

[0014] As described above, in the present invention, a specimen is stained with a plurality of staining methods, and multiple images of the specimen are obtained by taking images multiple times during the process. Then, from at least one of the specimen images, an internal region inside the cell membrane is first identified, and then the cell region is determined by expanding the region outward. Therefore, even if the cell density of the specimen is high, individual cells can be accurately identified, and it is also possible to handle specimens that are multiplexed immunostained. [Brief description of the drawings]

[0015] [Figure 1] 1 is a flowchart showing the flow of a sample analysis process according to the present embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a method for imaging a specimen. [Diagram 3] FIG. 2 is a diagram illustrating the principle of registration processing. [Figure 4] 13 is a flowchart showing a process of extracting a cell region. [Diagram 5] FIG. 1 is a first diagram illustrating a schematic change in an image during processing. [Figure 6] FIG. 2 is a second diagram showing a schematic diagram of changes in an image during processing. [Figure 7] 10A and 10B are diagrams illustrating a process of expanding an internal region. [Figure 8] FIG. 1 is a diagram illustrating an example of a machine learning model that can be used in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a specimen analysis process (hereinafter simply referred to as "analysis process") which is one embodiment of the specimen analysis method according to the present invention will be described. This analysis process is a process for identifying the position of each cell from an image of a specimen containing a large number of cells, such as a pathological tissue specimen, and acquiring quantitative information such as the expression state and expression level of a biological substance in each cell. The basic technical concept of this process follows the technique of multiple immunostaining, and since the basic principle of multiple immunostaining is well known, a description thereof will be omitted. First, an overview of this analysis process will be described with reference to FIG. 1, and then the contents of each processing step will be described in detail.

[0017] 1 is a flowchart showing the flow of the specimen analysis process of this embodiment. First, a specimen of pathological tissue to be analyzed is prepared (step S101). For example, the specimen can be prepared by fixing a tissue slice on a glass slide.

[0018] The specimen thus prepared is subjected to a staining process using a staining method according to the purpose (step S102), acquisition of specimen image data by capturing an image after staining (step S103), and destaining process (step S104) in this order, which are repeated multiple times. More specifically, each time the specimen is stained, the specimen is imaged using an imaging method described below and digitized as specimen image data. After the image is captured, the specimen is destained in preparation for the next staining. The above process is repeated until imaging using all of the previously prepared staining methods is completed (step S105). As a result, specimen image data corresponding to multiple specimen images with different staining states is acquired for the same specimen.

[0019] Since decolorization and staining are performed between images of the specimen captured by such imaging, the position of the specimen may be shifted within the image frame even when the images are captured by the same imaging device. In order to correct such shifts and align the images, a registration process is performed (step S106).

[0020] Using the multiple specimen images in which the mutual positional deviations have been corrected, a cell region extraction process is performed to identify the region occupied by each individual cell in the specimen (step S107). This identifies the position of each cell in the specimen image and the region it occupies. As a result, even if a large number of cells are densely packed in the specimen, the cells are recognized as separate.

[0021] A quantitative analysis by image cytometry is performed on each of the identified cells (step S108). As a result, quantitative information such as the state and expression amount of the biological material expressed in each cell is obtained for each cell. The obtained information is statistically processed as necessary and provided for evaluation by the user (clinician, researcher, etc.). Various image cytometry techniques have been put into practical use, and in this embodiment, an appropriate one selected from those depending on the purpose can be applied, so a detailed description of the contents of image cytometry will be omitted.

[0022] Of the above-mentioned processing steps, steps S106 to S108 can be executed as image processing by a computer device, for example. In this sense, the present invention can be embodied as a program for causing a computer device to execute these steps, or as a recording medium on which the program is non-temporarily recorded.

[0023] Next, the details of each process step in the above-mentioned analysis process will be described in order. First, a commonly used method can be applied to the preparation of the specimen in step S101. However, in order to separate each cell in the specimen image, it is preferable that the tissue slice to be the specimen is composed of a single layer of cells. In other words, it is preferable that the thickness of the specimen is equal to or smaller than the thickness of the cells that compose the tissue. Specifically, it is preferable that the thickness of the specimen is thinner than 4 μm, which is a common thickness for tissue specimens used in immunohistochemical analysis, and can be, for example, 2 μm or less. By using a thin tissue slice, the overlap of cells in the thickness direction of the specimen is reduced, the clarity of nuclear staining such as membrane staining, cytoplasmic staining, and nuclear staining is improved, and the accuracy of identifying the cell region can be improved. While making the specimen thin in this way is advantageous for cell recognition, it is disadvantageous for multiple staining because the specimen is easily damaged and the number of stainings cannot be increased. For this reason, it is preferable to limit the number of stainings to the minimum number required to perform the intended analysis.

[0024] The staining process in step S102 can be selected from various staining methods that are in practical use according to the purpose. For the purpose of performing immunohistochemical analysis, it is preferable to include at least one staining method that utilizes an antigen-antibody reaction. For example, depending on the purpose of the analysis, it is possible to stain proteins (membrane-localized proteins, nuclear-localized proteins, cytoplasm-localized proteins, etc.) that are effective for classifying cell types and functions. In the field of immunity, which has recently been actively researched, membrane-localized proteins are often used to distinguish cell types and maturity. In addition, from the viewpoint of accurately identifying cell regions, it is preferable to include staining for membrane-localized proteins or cytoplasm-localized proteins.

[0025] As described later, in this embodiment, an internal region, which is a region inside the cell membrane, is extracted from a cell in a specimen image. For this purpose, it is desirable to include at least one of a staining method for selectively staining biological material inside the cell membrane and a staining method for selectively staining only the cell membrane. For example, a cell nucleus, a cytoplasm, etc. can be used as the biological material inside the cell membrane. For this purpose, it is desirable to include a staining method for selectively staining the cell nucleus and / or the cytoplasm. Furthermore, hematoxylin-eosin (HE) staining, which is a common staining method for bright-field cell observation, or an alternative staining method thereof may be included so that the image can be visually observed by a user. HE staining can stain the cell nucleus and cytoplasm blue and red, respectively, and is therefore also effective for extracting the internal region described above.

[0026] When the type of cells to be analyzed is predetermined, it is effective to stain a protein that has been confirmed to be expressed or is expected to be expressed in the cells. For example, staining LCA (leukocyte common antigen) protein is effective for analyzing immune cells. Also, staining cytokeratin is effective for analyzing epithelial tumor cells.

[0027] By performing multiple immunostaining, which combines immunostaining for multiple antigens, it becomes possible to visualize multiple types of biological material individually, and it is suitable for investigating their colocalization, for example. The following are examples of staining methods that can be applied to such multiple immunostaining, and these can be used in appropriate combinations. For example, in tumor immunity research, immunostaining for antigens and proteins such as CD3, CD4, CD8, CD20, CD79a, and PD-1, which plays a role in the immune mechanism, is used. Then, the types and states of T cells and B cells are analyzed based on the combined positive / negative information. In addition, since there is no immunostaining that ideally stains all cells equally, it is possible to improve the accuracy of identifying cellular regions by combining multiple immunostaining images.

[0028] In step S103, the stained specimen is imaged to convert it into digital data. In the process shown in FIG. 1, an image is taken for each staining, but depending on the purpose, it may include a case where an image is taken once after a plurality of types of staining are performed in sequence. Specifically, the image can be taken as follows.

[0029] Fig. 2 is a diagram showing an example of a method for imaging a specimen. Fig. 2(a) shows an example of the main configuration of an imaging device 1 that realizes imaging. For example, a specimen S prepared on a slide glass is supported in a substantially horizontal position by an appropriate supporting means such as a contact glass 11. An illumination light source 12 is disposed above the specimen S, and causes substantially uniform illumination light L to be incident on the specimen S. If the illumination light L is visible light, bright field imaging is possible, and if it is excitation light for a fluorescent substance, fluorescent imaging is possible.

[0030] An image scanner 13 is disposed below the specimen S supported by the contact glass 11. The image scanner 13 has a one-dimensional (linear) image sensor (not shown) whose longitudinal direction is perpendicular to the paper surface. As indicated by the dotted arrow, the image scanner 13 moves in a direction perpendicular to the longitudinal direction, i.e., in the lateral direction of the paper surface, to scan the specimen S, thereby acquiring digital image data corresponding to a two-dimensional image of the specimen S. Imaging may be performed using a two-dimensional image sensor.

[0031] FIG. 2(b) shows an example of a specimen image. As shown in FIG. 2(b), the specimen image Is is captured so that the entire tissue slice T in the specimen fits within the field of view. Such an image is sometimes called a virtual slide. A large tissue slice T may be divided into multiple images and captured. It is desirable that the imaging resolution is sufficiently finer than the size of a cell. In this sense, an imaging device that uses a high-resolution linear image scanner to scan the specimen S to capture an image is effective. It is desirable that the acquired images can be presented to the user at various display magnifications.

[0032] In step S104, the specimen after imaging is decolorized. In this way, it is possible to eliminate the influence of the previous staining before a new staining. For example, it is possible to obtain the results of each staining individually for biological materials that develop color using two different staining methods. Also, for example, it is possible to distinguish between multiple types of biological materials that develop color using one staining method using the results of the other staining method. Note that if the influence of the previous staining is not a problem, the next staining may be performed without decolorization.

[0033] Multiple immunostaining visualizes multiple biological materials in a single image by superimposing multiple staining results on a specimen. Although destaining makes it impossible to obtain such a superimposing effect, images of the specimen in each staining state are captured and saved as specimen image data. Therefore, it is possible to simulate the effect of multiple immunostaining by synthesizing these specimen images through image processing. Rather, since images of any staining state can be selected and superimposed regardless of the order of staining, it is also possible to create a pseudo-stained image that cannot be achieved by multiple immunostaining on a specimen.

[0034] Since each specimen image is captured while repeatedly staining and decolorizing, the positions of the tissue slices T may be misaligned between images due to variations in the setting position of the specimen S during imaging. To address this problem, in step S106, the specimen images acquired by each imaging are aligned (registered). The registration is performed by utilizing corresponding points between the specimen images.

[0035] FIG. 3 is a diagram showing the principle of registration processing. Here, registration between three specimen images Is1 to Is3 of the same specimen, stained differently from one another, is taken as an example. Corresponding points between specimen image Is1 and specimen image Is2 are designated by symbols P and P', respectively. Here, "corresponding points" means that one point in one image and one point in the other image are in a relationship that represents the same position on the specimen S. Similarly, corresponding points between specimen image Is2 and specimen image Is3 are designated by symbols Q and Q', respectively.

[0036] Since the staining methods are different between the specimen images Is1 to Is3, it may be difficult to identify biological materials that are visualized in one image because they are not visualized in the other images. For example, there is a case where a point corresponding to point Q does not appear in the specimen image Is1, and a point corresponding to point P does not appear in the specimen image Is3. However, as illustrated below, if each specimen image has a corresponding point with at least one other specimen image, it is possible to indirectly correct between images where corresponding points cannot be found by performing position correction between them.

[0037] In the example of FIG. 3, point P in specimen image Is1 and point P' in specimen image Is2 correspond to each other. Also, point Q in specimen image Is2 and point Q' in specimen image Is3 correspond to each other. When these specimen images Is1 to Is3 are superimposed without registration, as shown in the lower left of FIG. 3, in the composite image Im, due to the above-mentioned positional shift during imaging, the positions of at least points P and P' and points Q and Q' may differ from each other. Such positional shift may be a cause of error when identifying the position of each cell from the specimen image.

[0038] Therefore, as shown in the lower right of FIG. 3, the positions of the images are corrected so that points P and P' overlap each other between the specimen images Is1 and Is2, and points Q and Q' overlap each other between the specimen images Is2 and Is3, and then the images are synthesized. Then, points P and P' coincide with each other on the synthesized image Im, and points Q and Q' coincide with each other. This makes it possible to eliminate the cause of errors when identifying the positions of individual cells comprehensively from multiple specimen images. As a specific correction method, a method can be used in which the position coordinates of corresponding points in multiple specimen images are obtained for each specimen image, and the relative positions between the images are shifted according to the difference between the coordinates. It is also possible to use a method that uses pixel information of the entire image without using the position coordinates of corresponding points, for example, a method of obtaining a relative position that reduces the difference in pixel values ​​at the same position, or a method of obtaining a relative position that maximizes the correlation value of pixels. The above is the principle of registration processing.

[0039] Next, the cell region extraction process in step S107 will be described with reference to Figures 4 to 8. The shapes and sizes of cells in a specimen vary greatly from one to another, and many cells are often in close contact with one another. The extraction process of this embodiment described below can accurately extract individual cells even from such specimens.

[0040] FIG. 4 is a flow chart showing the extraction process of the cell region. Also, FIG. 5 and FIG. 6 are diagrams showing the change of the image in the process. FIG. 5(a) shows a schematic example of an image in which a plurality of cells are in close contact with each other. Here, four cells C1 to C4 are in contact with each other. Each of the cells C1 to C4 has a nucleus (cell nucleus) N1 to N4 inside. The periphery of each of the cells C1 to C4 is covered with a cell membrane M, but if the cells are in contact with each other, the boundary may become unclear. Cytoplasm Cy exists between the cell nuclei C1 to C4 and the cell membrane M. Although not shown in the figure, various biological materials such as various organelles (mitochondria, Golgi apparatus, etc.), cytoplasmic matrix, and vacuoles may be included in the cytoplasm Cy1 to Cy4.

[0041] It is necessary to separate and identify individual cells from such a cell group. To make this possible, in this embodiment, first, at least one of a plurality of specimen images stained by different staining methods is used to extract the inside of the cell in the specimen image, more specifically, the area inside the cell membrane M (referred to as the "internal area" in this specification) (step S201). For example, if there is a specimen image to which a staining method that selectively stains biological material that exists only inside the cell membrane is applied, that specimen image can be used. Also, for example, in an image in which only the cell membrane is selectively stained, the closed area surrounded by the cell membrane can be regarded as the internal area, so such an image can be used as the specimen image.

[0042] In addition, in extracting the internal region, a plurality of specimen images with different staining conditions may be used. For example, when two or more types of biological material are stained with one staining method, it may be impossible to extract the desired biological material from only the specimen image. By combining a plurality of specimen images with different staining conditions, such a problem may be solved. Specifically, for example, a specimen image in which the cell nucleus is stained and a specimen image in which the cytoplasm is stained may be combined to extract the internal region. Also, for example, a region surrounded by the cell membrane and containing the cell nucleus inside may be extracted as the internal region from a specimen image in which only the cell membrane is stained and a specimen image in which the cell nucleus is stained.

[0043] In addition, any of various biological materials present inside the cell membrane within the cell can be used as the extraction target. For example, any of the following can be used: cell nuclei, cytoplasmic matrix, mitochondria, Golgi apparatus, and other cytoplasm. In order to perform extraction with high accuracy, it is desirable to select a biological material that has the property of being selectively stained by a certain staining method. For example, cell nuclei are suitable biological materials for such purposes, since various methods have been established for selectively staining only cell nuclei.

[0044] It should be noted that the biological material contained in the internal region does not necessarily have to be one type. For example, in a specimen image in which two or more biological materials are simultaneously expressed, if it is clear that all of these biological materials are present within a cell, it is not necessary to distinguish between these biological materials. In other words, multiple types of biological materials may be extracted as internal regions without being distinguished from one another.

[0045] FIG. 5(b) shows a schematic diagram of the results of extracting the regions corresponding to the cell nuclei N1-N4 and their surrounding cytoplasm Cy1-Cy4 from the image in FIG. 5(a). Here, the region consisting of the cell nucleus N1 and its surrounding cytoplasm Cy1 can be regarded as the "internal region" R1 of one cell C1. Similarly, the region consisting of the cell nucleus N2 and its surrounding cytoplasm Cy2 corresponds to the "internal regions" R2-R4 of cell C2, the region consisting of the cell nucleus N3 and its surrounding cytoplasm Cy3 corresponds to the "internal regions" R2-R4 of cell C3, and the region consisting of the cell nucleus N4 and its surrounding cytoplasm Cy4 corresponds to the "internal regions" R2-R4 of cell C4.

[0046] FIG. 5(c) shows a schematic example of the structure of a small cell such as a lymphocyte. In small cells C5-C11 such as lymphocytes, for which multiple immunostaining is more effective, cytoplasmic Cy is scarce, and the inner region of the cell membrane can be considered to be occupied almost entirely by the cell nuclei N5-N11. In such cases, it is considered that there will be no significant error even if the region occupied by the cell nuclei N5-N11 in the specimen image is considered to be the "internal region". That is, as shown in FIG. 5(d), the region of the cell nuclei N5-N11 extracted from the image can be directly used as the internal region R5-R11. Therefore, it is also possible to extract the internal region from a specimen image in which only the cell nuclei are stained.

[0047] The internal regions R1 to R4 (or internal regions R5 to R11) extracted in this way do not include the region of the surrounding cell membrane M, so that even if the cells are in contact with each other through the cell membrane M in the specimen, they are isolated from each other. That is, each of the internal regions R1 to R4 (R5 to R11) corresponds one-to-one with each of the cells C1 to C4 (C5 to C11). By such correspondence, the cells C1 to C4 (C5 to C11) are recognized as separate from each other even if they are in close contact, and thereafter, each of the cells C1 to C4 (C5 to C11) can be treated individually. Therefore, the identified internal regions C1 to C4 (C5 to C11) are labeled in order to distinguish them (step S202). This labeling can also be used to distinguish each of the cells C1 to C4 (C5 to C11) in the analysis process on a single cell basis.

[0048] The internal regions R1-R4 (R5-R11) indicate that at least the regions are contained within the cells, but do not indicate the outer edge of the cells including the cell membrane M. Therefore, in this embodiment, image processing is performed to expand the contours of the internal regions R1-R4 (R5-R11) that have already been separated from each other outward (step S203). FIG. 6(a) shows the extraction result of FIG. 5(b), and FIG. 6(b) shows the result of the expansion process based on the extraction result of FIG. 5(d), with the dotted lines showing an example of the contours after expansion. The expanded region, that is, the region that combines the original internal region and its expanded portion, is finally identified as the cell region corresponding to each of the cells C1-C4 (C5-C11) (step S204).

[0049] FIG. 7 is a diagram showing a schematic diagram of the process of expanding an internal region. Here, the expansion of internal regions R1, R2 near the junction of two cells C1, C2 is explained as an example. As shown by the arrows in FIG. 7(a), the contour parts corresponding to the outer edges of internal regions R1, R2 are expanded outward by a predetermined amount (e.g., one pixel). As a result, internal regions R1, R2 spread by a predetermined amount toward the region corresponding to the cell membrane M surrounding them. The dotted lines in FIG. 7(a) show the contours after the internal regions R1, R2 have been expanded.

[0050] As shown in FIG. 7(b), such contour expansion is performed until the width (expansion width) D of the expanded portion reaches a predetermined value or until the expanded contour comes into contact with another (expanded) internal region. The specified value for the expansion width D can be preset as a value corresponding to the standard thickness that the cell membrane of the cell can have, depending on the type of cell. As shown in FIG. 5(d), when only the cell nucleus region is extracted as the internal region, the sum of the standard thicknesses of the cytoplasm and cell membrane around the cell nucleus can be set as the specified value.

[0051] In this way, the expanded internal region R1a becomes a combination of the original internal region R1 and a region of a certain width surrounding it, which roughly coincides with the region occupied by each cell including the cell membrane M. Even in the area where two cells contact each other, a boundary is defined at an intermediate position between the two cells, as shown by the dashed line in Fig. 7(b).

[0052] By carrying out the expansion process in this way, as shown in FIG. 6, it is possible to specify for each cell the regions R1a-R4a (R5a-R11a) that roughly coincide with the regions occupied by each of the cells C1-C4 (C5-C11) in the specimen image. In this sense, the internal regions R1a-R11a after expansion represent the regions occupied by each of the cells C1-C11 in the image, that is, the cell regions. Even if adjacent cell regions are in contact with each other, they are already distinguished as individual regions. By cutting out the regions corresponding to the cell regions from the original image, it is possible to cut out the cells C1-C4 (C5-C11) from the image. In this way, the individual cells are clearly distinguished, and their respective contours are appropriately defined.

[0053] In this method, the specified value for the expansion amount is determined based on the thickness of the cell membrane in an isolated cell, so that the area occupied by the cell can be appropriately specified even for an isolated cell. In addition, when it is clear that a large number of cells are closely spaced in a specimen, the specified value may be determined based on the thickness of the cell membrane in such a case.

[0054] By identifying the cell area occupied by each cell in this manner, even if multiple cells are densely packed together, it is possible to measure each cell individually and obtain its quantitative information by measurement. That is, by performing image cytometry (step S108 in FIG. 1), various information such as the position, size, and shape of each cell, and the expression state and expression amount of a biological substance can be obtained. In this embodiment, the contents of image cytometry and the type of information obtained thereby are not particularly limited, but the results can be used, for example, to present the staining property of immunostaining, or to judge whether a specific disease is positive or negative.

[0055] A learning model constructed by machine learning can be used to extract internal regions from specimen images and to analyze individual cells with specified cellular regions. In specimens containing many cells, individual differences between cells are large, making it difficult to extract individual cells by simple threshold processing or pattern matching. By using a learning model that performs appropriate machine learning based on the morphological characteristics of cells, it is possible to realize analysis processing that can flexibly respond to such individual differences.

[0056] FIG. 8 is a diagram showing an example of a machine learning model that can be used in this embodiment. Here, as an example, construction of a learning model that can be used when extracting an internal region corresponding to a region that combines a cell nucleus and a cytoplasm from a specimen image is described. However, a similar method can also be applied when the extraction target is another biological material, or when the purpose is to detect and measure a desired biological material from each cell after the cell region is specified.

[0057] FIG. 8(a) is a flowchart showing a process for constructing a learning model, and FIG. 8(b) is a diagram showing a model of the learning model. First, a user collects a large number of typical images Ia that serve as teachers in machine learning (step S301), and prepares corresponding correct images Ib (step S302). The correct images Ib are images for teaching the area occupied by the biological material to be extracted in the typical images Ia. For the purpose of extracting the internal area inside the cell membrane from the image of a cell, various stained images such as membrane staining, cytoplasm staining, and nuclear staining are appropriate as the typical images Ia, and images that clearly show the internal area corresponding to the stained image to be extracted are appropriate as the correct images Ib. Correct images Ib are prepared for each of the collected typical images Ia, and teacher data Dt is created by pairing the typical images Ia with the corresponding correct images Ib (step S303).

[0058] The teacher data Dt is provided to a learning model 100 having an appropriate learning algorithm, and the learning model 100 executes machine learning (step S304). Various learning models have been proposed, and an appropriate one can be selected and applied in this embodiment. In order to construct a highly versatile learning model that can handle a variety of cells and biological materials, for example, a deep learning algorithm can be suitably used.

[0059] The learning model 100 constructed in this manner has the following functions. That is, when a test image It including a cell is provided as an input image to the learning model 100, the learning model 100 outputs an output image Io that represents only the region inside the cell membrane M of the cell Ct in the test image It, that is, the internal region Ro. That is, the learning model 100 has the function of extracting only the internal region Ro from the unknown test image It. By providing appropriate teacher data according to the desired extraction target, the learning model 100 has the function of extracting various biological materials from the test image.

[0060] In this embodiment, in which multiple specimen images with different staining conditions are used, region extraction using the learning model as described above may be performed on each individual specimen image, or region extraction using the learning model may be performed on an image obtained by combining multiple specimen images. Even when region extraction is performed using a combined image, it can be said to be "extraction based on the original specimen image" in that information contained in the original specimen image is used.

[0061] As described above, in this embodiment, the specimen S to be evaluated is stained in a number of different ways in sequence, and the specimen S is imaged each time. Then, based on a specimen image in which at least one of the stains, for example the cell nucleus, cytoplasm, or cell membrane, is stained, an internal region inside the cell membrane is extracted. The internal region thus extracted does not include the cell membrane region. Therefore, even if cells are in contact with each other, the internal regions extracted from each cell are isolated from each other. In other words, each extracted internal region indicates the presence of one cell. This allows each cell in the specimen to be individually identified, even if they are in contact with each other.

[0062] By expanding the internal region outward, it is possible to include the surrounding cell membrane region. The amount of expansion is set according to the possible thickness of the cell membrane. This makes it possible to identify the outline of the cell, including the cell membrane, while maintaining the independence of each individual cell. Once the area occupied by each cell in the specimen image has been identified, purpose-specific measurements (image cytometry) can be performed on each cell to obtain quantitative information about each individual cell.

[0063] In this manner, in this embodiment, even when a specimen contains many cells and the cells are in contact with each other, it is possible to obtain quantitative information on a cell-by-cell basis by identifying each cell individually and performing the necessary measurements. In other words, single-cell analysis is possible. Various combinations of multiple types of staining can be applied, and therefore multiple immunostaining can also be performed.

[0064] The present invention is not limited to the above-mentioned embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, in the above-mentioned embodiment, the region corresponding to the cell nucleus and the region corresponding to the cytoplasm are used as the "internal region" of the present invention, but the present invention is not limited to this. That is, it is possible to stain biological material located inside the cell membrane using at least one of membrane staining, nuclear staining, and cytoplasm staining, or to stain only the cell membrane, and to detect the biological material from the specimen image to extract the internal region.

[0065] In the above embodiment, a learning model employing deep learning as a machine learning algorithm is used for cell analysis, and learning is performed using pairs of a typical image and a correct answer image as training data. However, the learning algorithm and the training data provided thereto are not limited to these, and various machine learning algorithms can be applied as long as they can be used for image region segmentation. In addition, if the target biological material can be extracted by other image processing such as threshold processing or pattern matching processing, an extraction method other than machine learning may be used in combination.

[0066] Furthermore, in the above embodiment, registration is performed between a plurality of specimen images. However, if imaging is possible in which the positional deviation between each specimen image is negligible, registration can be omitted.

[0067] In the above embodiment, after the cells in the specimen are individually identified, measurement (image cytometry) is performed for each cell. However, it is also possible that, for example, a certain device performs up to the extraction process of the cell region to identify each cell, and outputs the identification result to the outside, and the measurement is performed by another device.

[0068] As described above by way of example and specific embodiments, in the specimen analysis method according to the present invention, the specimen is preferably a tissue section having a thickness of less than 4 μm, more preferably 2 μm or less. In such a specimen, erroneous recognition of individual cell regions due to overlapping of cells in the thickness direction can be reduced.

[0069] Furthermore, for example, if at least one of the multiple stains selectively stains biological material present only inside the cell membrane, it is possible to extract an internal region from a specimen image captured after the staining. The internal region thus extracted corresponds to biological material present inside the cell membrane, and therefore is highly likely to be the inside of the cell.

[0070] For example, from a specimen image in which cell nuclei are stained, Surrounded by the cell membrane and The area including the cell nucleus can be extracted as the internal area. The cell nucleus is located closer to the cell membrane than the cell membrane. Inside The cell nucleus is a representative biological material present near the surface of a cell, and since the cell nucleus is a relatively large structure within a cell, and various methods for selectively staining the cell nucleus have been established, the cell nucleus is therefore suitable as a biological material that indicates the presence of an individual cell.

[0071] Also, for example, at least one of the multiple types of staining may selectively stain the cell membrane, and the internal region may be extracted from a specimen image captured after the staining. In this case, a region including a cell nucleus may be extracted as the internal region from a specimen image in which the cell membrane is stained and a specimen image in which the cell nucleus is stained. In a specimen image in which only the cell membrane is stained, a closed region surrounded by the stained cell membrane can be regarded as the inside of a cell. Therefore, it is possible to extract the internal region using a specimen image in which the cell membrane is selectively stained. In particular, by using the specimen image in which the cell nucleus inside the cell is stained in combination, the internal region of the cell can be extracted more accurately.

[0072] For example, image processing for expanding each point on the contour of the internal region toward the outside of the internal region is performed by A predetermined value is determined according to the thickness of the cell membrane in the image. The process is continued until a specified value is reached or the expanded internal region comes into contact with another expanded internal region, and the region obtained by combining the internal region and the expanded region by the image processing is regarded as the cell region. According to this configuration, by appropriately setting the specified value, the extracted internal region is expanded to the region of the cell membrane surrounding it, and a cell region that generally coincides with the region occupied by the actual cell can be identified.

[0073] For example, the extraction of the internal region can be performed by inputting a specimen image into a learning model previously constructed by machine learning and acquiring the output image. Since the cells in a specimen vary greatly from one another, a learning model previously machine-learned using a large number of cases can be suitably used as a method for extracting a specific biological material with high accuracy even in such cases. It is also possible to construct a highly versatile learning model that can be applied to various types of cells.

[0074] Also, for example, the positioning may be performed using the corresponding positions between the multiple specimen images, and the internal region may be extracted based on the aligned multiple specimen images. When staining and imaging are performed repeatedly, it is inevitable that the specimen will be misaligned with each imaging. In order to reduce measurement errors caused by this misalignment, it is effective to perform the positioning based on the positional relationship between the corresponding positions between the multiple specimen images.

[0075] For example, after imaging, the specimen may be destained before the next staining. With this configuration, the influence of the previous staining result on the subsequent staining and imaging can be eliminated. Also, if the specimens that are not influenced by the previous staining are individually imaged, it is possible to generate a pseudo multiple immunostained image by combining the images. [Industrial Applicability]

[0076] The present invention is useful for analyzing specimens containing a large number of cells, and is particularly suitable for use in immunohistochemical analysis of pathological specimens. [Explanation of symbols]

[0077] 1. Imaging device 11 Contact Glass 12 Lighting source 13 Image Scanner 100 Learning Models C1~C4 cells Cy,Cy1~Cy4 Cytoplasm Is, Is1~Is3 Sample images M cell membrane N,N1~N11 cell nucleus R1~R11, Ro internal area R1a~R11a: Expanded inner region (cell region) S specimen

Claims

1. A step of sequentially staining a specimen to be evaluated with a plurality of types of staining and capturing images of the specimen after at least two stainings to obtain a plurality of specimen images; extracting a closed region surrounded by the cell membrane as an internal region of the cell in the specimen, the internal region being inside the cell membrane, based on at least one of the specimen images; determining cellular regions corresponding to individual cells contained in the specimen by expanding the extracted internal regions outward; performing image cytometry on the cellular region for a plurality of the specimen images; A sample analysis method comprising:

2. The specimen analysis method according to claim 1 , wherein at least one of the plurality of stains selectively stains a cell membrane, and the internal region is extracted from an image of the specimen captured after the staining.

3. 3. The method for analyzing a specimen according to claim 1, wherein the specimen is a tissue section having a thickness of less than 4 μm.

4. The method for analyzing a specimen according to claim 3, wherein the thickness of the tissue section is 2 μm or less.

5. 5. A specimen analysis method according to claim 1, wherein at least one of the plurality of stains selectively stains biological material present only inside a cell membrane, and the internal region is extracted from the specimen image captured after the staining.

6. A specimen analysis method according to any one of claims 1 to 5, wherein image processing is performed to expand each point on the contour of the internal region toward the outside of the internal region, until the amount of expansion at that point reaches a predetermined value determined according to the thickness of the cell membrane in the image or until it comes into contact with another expanded internal region, and the combined area of ​​the internal region and the area expanded by the image processing is defined as the cell region.

7. The specimen analysis method according to claim 1 , wherein the extraction of the internal region is performed by inputting the specimen image into a learning model previously constructed by machine learning and acquiring an output image thereof.

8. 8. The specimen analyzing method according to claim 1, further comprising: performing registration between the plurality of specimen images by utilizing corresponding positions between the plurality of specimen images; and extracting the internal region based on the plurality of registered specimen images.

9. 9. The method for analyzing a specimen according to claim 1, further comprising the step of: destaining the specimen after the imaging and before the next staining.

10. 1. An image processing method using a plurality of specimen images obtained by sequentially staining a specimen to be evaluated with a plurality of types of staining and capturing images of the specimen after at least two of the stainings, comprising: extracting a closed region surrounded by the cell membrane as an internal region of the cell in the specimen, the internal region being inside the cell membrane, based on at least one of the specimen images; determining cellular regions corresponding to individual cells contained in the specimen by expanding the extracted internal regions outward; performing image cytometry on the cellular region for a plurality of the specimen images; An image processing method comprising:

Images