Cell counting method, method for constructing machine learning model for cell counting, program and recording medium

Abstract

To automatically and accurately count the cell number from an image obtained by imaging a cell lump.SOLUTION: A cell counting method according to the present invention comprises the steps of: acquiring an image stack including a plurality of images obtained by imaging a cell lump in a bright field at mutually-different focal depths; creating a plurality of heat maps respectively corresponding to the plurality of images by using a machine learning model; detecting a peak from each heat map; associating peaks which belong to mutually-different heat maps among the detected peaks and have a distance in the direction along the map plane and a distance in the focal depth direction smaller than a prescribed value with each other; and counting the number of peaks by regarding the associated peaks as one peak. The machine learning model is constructed with machine learning using a teacher image being a bright field image of a cell and a correct answer image being the heat map given with the weight that becomes larger as getting closer to the cell center as teacher data.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for automatically counting the number of cells constituting a cell cluster by analyzing an image of the cell cluster. [Background technology]

[0002] In the technical fields of pathology and cell culture, the number of cells present in an image of a cell mass consisting of multiple cells is often used as a criterion for judging the state of cells or tissues, such as the developmental state. Since counting a large number of cells visually places a heavy burden on the operator, a technology has been proposed for automatically counting the number of cells by analyzing the image.

[0003] For example, in the technology described in Patent Document 1, a sample is stained with a staining method that stains cell nuclei, and the shape of the observed object is grasped from the brightness of the image obtained by microscopic imaging, and when micronuclei are detected, the number of cells is determined by counting the nuclei after corresponding them according to their distance from the main nucleus. In addition, in the technology described in Patent Document 2, for example, in order to distinguish cells that are close to each other and have unclear boundaries, an image of a fluorescently labeled sample is used to calculate a score for a pixel of interest from the pixel values ​​of multiple pixels in an area including the pixel of interest, and the position or number of individual cells is determined based on the score. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2001-269195 A [Patent Document 2] International Publication No. 2005 / 057496 Summary of the Invention [Problem to be solved by the invention]

[0005] When the cell mass to be observed has a three-dimensional structure and multiple cells overlap in the depth direction of the image, it is difficult to separate and count individual cells using the method based on the brightness value of the image as in the conventional technology described above. For example, in the observation of a fertilized egg (embryo), many cells are densely packed in a spherical shape, and each cell is almost transparent. For this reason, it is very difficult to separate and count them individually from the brightness value of a single microscope image captured by receiving transmitted light or reflected light.

[0006] For this reason, there is a need to establish a technology that can more accurately determine the number of cells that compose a cell mass having a three-dimensional structure. In addition, in the above-mentioned conventional technology, the cells are stained or fluorescently labeled, and therefore cannot be applied to counting live cells. Therefore, it is more preferable to be able to count cells using images that are not based on such invasive imaging methods.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique that makes it possible to automatically and accurately count the number of cells from an image of a cell cluster. [Means for solving the problem]

[0008] One aspect of the present invention is a cell counting method for counting the number of cells constituting a cell cluster, and in order to achieve the above object, the cell cluster is photographed in a bright field at different focal depths. Complex Contains a number of images 、The method includes the steps of acquiring an image stack, creating a plurality of heat maps corresponding to the plurality of images using a machine learning model, detecting peaks from each of the heat maps, associating the detected peaks that belong to different heat maps and have a distance along the map plane and a distance along the focal depth direction that are each smaller than a predetermined threshold, and counting the number of the associated peaks by regarding the associated peaks as one peak. Here, the machine learning model is a teacher image that is a bright-field image of a cell, and a region inside the contour of a cell focused in the teacher image. In the above outline Larger than the outside Kuka It was constructed through machine learning using a set of training data, which is a heat map in which the weights are increased the closer the image is to the center of the cell, and a ground truth image.

[0009] In the invention configured in this way, the number of cells is counted based on so-called multi-focus images obtained by capturing bright-field images of the cell cluster multiple times at different focal depths. In bright-field images of a cell cluster made up of nearly transparent cells, it is expected that not only the cells appearing on the surface of the cell cluster, but also the cells behind them will be captured in a state of being focused on one of the multiple images by capturing images at different focal depths. Therefore, if the cells that are in focus are extracted from each image, they will be considered to cover the entire cell cluster.

[0010] However, due to the relationship between the depth of field of the imaging optical system, the pitch of change in focal depth during imaging, the size of the cell, etc., it is possible that one cell is in focus in multiple images. For this reason, simply counting the cells detected from each image may result in duplicate counts, making it impossible to obtain accurate counting results. The present invention aims to solve this problem by using the following configuration.

[0011] First, cells in each image are identified by detecting peaks in the heat map output by the machine learning model. In the heat map, the closer to the center of the cell, the greater the weight is given, and the peak position corresponds to the center of the cell. Second, peaks detected in each image that are close to each other are counted as one peak.

[0012] Simply identifying the area occupied by a cell in each image makes it difficult to determine whether cell areas detected from multiple images correspond to the same cell or different cells when they overlap. On the other hand, for the purpose of determining the number of cells, it is not essential to accurately identify the area occupied by a cell in an image. Rather, a method of identifying the center position of the cell is more suitable, because it is possible to determine whether they originate from the same cell from the distance between the detected cell centers.

[0013] Specifically, if the size of the cells is known to some extent, it is possible to estimate the lower limit of the distance that their centers can take. Therefore, among the peaks corresponding to the cell centers detected in multiple images, those that are closer than the estimated lower limit can be considered to originate from the same cell. On the other hand, each isolated peak can be considered to correspond to one cell. In this way, if multiple peaks that are closer than a certain distance are considered as a single peak, the number of cells can be expressed by the number of peaks. This makes it possible to avoid counting the same cell multiple times.

[0014] In addition, detection of individual cells is performed by estimating the center position of the cell using a machine learning model constructed using an image containing a focused cell as a teacher image. Therefore, if machine learning is performed using an image of a cell that has not been subjected to an invasive method such as staining the cell nucleus as a teacher image, an image captured non-invasively can be used for the cell cluster to be counted.

[0015] Another aspect of the present invention includes a step of acquiring a plurality of teacher images, which are bright-field images of a cell cluster, and for each of the teacher images, determining whether the inside of the outline of a cell focused in the teacher image is a region of the cell cluster that ... In the above outline Larger than the outside Kuka The closer to the cell center the larger the weight but Give while less-clear cell images are given more weight. The present invention relates to a method for constructing a machine learning model for cell counting, the method comprising: a step of creating a ground truth image that is a pseudo heat map; and a step of executing machine learning using a set of the teacher image and the ground truth image as teacher data to construct a machine learning model. With the invention configured in this manner, it is possible to efficiently construct a machine learning model suitable for executing the above-mentioned cell counting method.

[0016] Another aspect of the present invention is a program for causing a computer to execute each step of the above-mentioned cell counting method or the method for constructing a machine learning model for cell counting. Another aspect of the present invention is a computer-readable recording medium non-temporarily storing the above-mentioned program. In the invention configured in this way, it is possible to use, for example, an existing computer device as the executing entity of the present invention. Effect of the Invention

[0017] As described above, according to the present invention, a heat map is created from each image of a multi-focus image stack, with the center of the cell being weighted more heavily, and the peaks are treated as the center of the cell, and multiple peaks whose inter-peak distance is less than a predetermined value are counted as being derived from the same cell. Therefore, even if the cell mass has a three-dimensional structure, the number of cells constituting the cell mass can be counted automatically and accurately. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an imaging device to which the cell counting method of the present invention can be applied. [Diagram 2]FIG. 2 is a diagram showing a schematic structure of an embryo serving as a live sample in this embodiment. [Diagram 3] 1 is a flowchart showing an overview of a cell counting process according to the present embodiment. [Figure 4] 1 is a flowchart illustrating a process for constructing a machine learning model. [Diagram 5] FIG. 11 is a diagram illustrating a process of creating a heat map. [Figure 6] FIG. 13 is a diagram illustrating an example of a method for detecting a peak point. [Figure 7] FIG. 13 is a diagram illustrating a process for aggregating peak points. [Figure 8] 13 is a flowchart showing a first aspect of adjacent peak aggregation processing. [Figure 9] FIG. 9 is a diagram illustrating an example of the adjacent peak aggregation process of FIG. 8. [Figure 10] 13 is a flowchart showing a second aspect of adjacent peak aggregation processing. [Figure 11] FIG. 11 is a diagram illustrating an example of the adjacent peak aggregation process of FIG. 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1 is a diagram showing a schematic configuration of an imaging device to which the cell counting method according to the present invention can be applied. This imaging device 1 is a device that images a living specimen such as cells held in a flat specimen container 10 called a dish, which has an open top. A predetermined amount of liquid as a culture medium M is poured into the specimen container 10, and a living specimen S such as cells cultured in this liquid under predetermined culture conditions becomes the subject of imaging by this imaging device 1. The culture medium may be one to which an appropriate reagent has been added, or may be one that gels after being poured in liquid form into the specimen container 10.

[0020] In this embodiment, the imaging subject is a living specimen, but the imaging subject of the present invention is not limited to this. For example, the imaging subject may be a tissue slice or a pathological specimen supported on an appropriate carrier. In addition, the sample container may be a so-called well plate having a plurality of depressions called wells, and the imaging subject may be a living specimen supported in each well.

[0021] The imaging device 1 includes a holder 11 for holding a sample container 10, an illumination section 12 arranged above the holder 11, an imaging section 13 arranged below the holder 11, and a control section 14 having a CPU 141 for controlling the operation of each of these sections. The holder 11 abuts against the peripheral portion of the lower surface of the sample container 10 to hold the sample container 10 in a substantially horizontal position.

[0022] The illumination unit 12 emits illumination light toward the sample container 10 held by the holder 11. A white LED (Light Emitting Diode), for example, can be used as a light source of the illumination light. A combination of the light source and an appropriate illumination optical system is used as the illumination unit 12. The illumination unit 12 illuminates the object to be imaged in the sample container 10 from above.

[0023] An imaging section 13 is provided below the sample container 10 held by the holder 11. In the imaging section 13, an imaging optical system is disposed directly below the sample container 10, and the optical axis of the imaging optical system is oriented vertically. Fig. 1 is a side view, and the up-down direction of the figure represents the vertical direction.

[0024] The imaging unit 13 captures a bright-field image of the object to be imaged in the sample container 10. Specifically, light emitted from the illumination unit 12 and incident on the liquid from above the sample container 10 illuminates the object to be imaged, and light transmitted downward from the bottom surface of the sample container 10 is incident on the light receiving surface of the imaging element 132 via an imaging optical system including an objective lens 131 of the imaging unit 13. An image (bright-field image) of the object to be imaged formed on the light receiving surface of the imaging element 132 by the imaging optical system is captured by the imaging element 132. The imaging element 132 is an area image sensor having a two-dimensional light receiving surface, and for example, a CCD sensor or a CMOS sensor can be used.

[0025] The imaging unit 13 can be moved in the horizontal and vertical directions by a mechanical control unit 146 provided in the control unit 14. Specifically, the mechanical control unit 146 operates the drive mechanism 15 based on a control command from the CPU 141 to move the imaging unit 13 in the horizontal direction, whereby the imaging unit 13 moves in the horizontal direction relative to the sample container 10. Also, the focus is adjusted by moving the objective lens 131 in the vertical direction.

[0026] When moving the imaging unit 13 in the horizontal direction, the driving mechanism 15 moves the illumination unit 12 integrally with the imaging unit 13 as shown by the dotted arrow in the figure. That is, the illumination unit 12 is disposed so that its optical center substantially coincides with the optical axis of the imaging unit 13, and moves in conjunction with the imaging unit 13 when the imaging unit 13 moves in the horizontal direction. As a result, even when the imaging unit 13 moves relative to the sample container 10, the optical center of the illumination unit 12 is always positioned on the optical axis of the imaging unit 13, and the illumination conditions for the imaging object are kept constant at any position, making it possible to maintain good imaging conditions.

[0027] The image signal output from the imaging element 132 of the imaging unit 13 is sent to the control unit 14. That is, the image signal is input to an AD converter (A / D) 143 provided in the control unit 14 and converted into digital image data. The CPU 141 executes appropriate image processing based on the received image data.

[0028] The control unit 14 further has a memory 144 for temporarily storing image data and data being calculated, and a storage 145 for storing and saving programs to be executed by the CPU 141 and data generated by the CPU 141. The memory 144 can be accessed at high speed by the CPU 144, but has a smaller storage capacity than the storage 145. The storage 145 is, for example, a hard disk drive (HDD), and has a large storage capacity but a slower access speed than the memory 144. These are used according to the purpose. The CPU 141 calls and executes the control programs stored in the storage 145 to perform various calculation processes described later.

[0029] Additionally, the control unit 14 is provided with an interface (IF) unit 142. The interface unit 142 has a user interface function of receiving operation input from the user and presenting information such as processing results to the user, as well as a function of exchanging data with an external device connected via a communication line. To realize the user interface function, the interface unit 142 is connected to an input receiving unit 147 that receives operation input from the user, and a display unit 148 that displays and outputs messages to the user, processing results, and the like.

[0030] Hereinafter, an embodiment of the cell counting method according to the present invention that can be performed using the imaging device 1 configured as above will be described. The imaging device 1 has a function of automatically counting the number of cells constituting a cell mass as a raw sample S by performing a cell counting process described later. Here, a process of imaging an early fertilized egg (hereinafter simply referred to as an "embryo") as the sample S and counting the number of cells constituting the embryo from the image will be described. Based on the data obtained in this way, it becomes possible to effectively support the evaluation work of the embryo by a user (specifically, a doctor or an embryologist). For example, in the culture of an embryo for the purpose of infertility treatment, the cell counting method of this embodiment can be applied to obtain knowledge for determining whether the culture is progressing well. Note that the processing target is not limited to such an embryo, and this process can be applied to various cell masses in which multiple cells are closely arranged.

[0031] 2 is a diagram showing a schematic structure of an embryo serving as a live sample in this embodiment. As is already known, when an egg is fertilized, cleavage begins, and a blastocyst is formed through a stage called a morula. The cell counting method of this embodiment is suitable for counting the number of cells that constitute an embryo in the morula stage immediately after fertilization, for example.

[0032] Figure 2(a) shows a schematic diagram of the structure of an embryo at an early stage (for example, from the 4-cell stage to the morula stage). The external shape of embryo E is roughly spherical. Its surface is covered with a layer Z of a jelly-like glycoprotein called the zona pellucida, and inside it are multiple cells C that are generated by cell division of the fertilized egg. When culture is progressing smoothly, as shown in the left diagram of Figure 2(a), the inside of the zona pellucida Z is occupied by relatively large cells C. As cleavage progresses, the number of cells C increases.

[0033] When bright-field imaging of such an embryo E is performed while varying the focal depth, indicated by the dotted line in the figure, a number of images are obtained, as shown diagrammatically in the right diagram of Fig. 2(a) . In image Ia, where the focal depth is the shallowest as viewed from the imaging unit 13 arranged below the specimen container 10, that is, where the focal position is positioned at the lowest position, none of the cells C constituting the embryo E are in focus, resulting in a blurred image.

[0034] The focal position moves upward, and in image Ib, for example, the cell C1 at the bottom is focused on, and the cell C1 becomes clear. In image Ic, where the focal position moves further upward, the out-of-focus cell C1 becomes blurred, and the cells C2 to C4, which are closer to the focal position, appear as clearer images. If cell C1 is nearly transparent, other cells behind it will appear in the image through cell C1. In image Id, where the focal position moves further upward, the out-of-focus cells C1 to C4 become blurred, and the images of cells C5 to C7 further up, which are closer to the focal position, appear through them.

[0035] In this way, by imaging the same sample multiple times in the same field of view while changing and setting the focal depth in multiple stages, a set of multiple images with the same field of view and different focal depths can be obtained, as shown in FIG. 2(b). This image set is called a multi-focus image stack. In each image in the image stack ST, the cells that make up the cell cluster (embryo E in this example) that are close to the focal position form clear images, while the images of the cells that are farther from the focal position become unclear. In addition, the light passes through the cells in front of the focal position (the side closer to the imaging unit 13) and the images of the cells behind them appear. In the cell number counting process of this embodiment, the number of cells that make up the cell cluster is counted by utilizing such characteristics of the image stack ST.

[0036] FIG. 3 is a flowchart showing an outline of the cell counting process of this embodiment. This process and each process described below are realized by the CPU 141 reading and executing a control program previously stored in the storage 145, and causing each part of the device to perform a predetermined operation. These various processes, except for the process of imaging the raw sample S, can also be realized by a computer device having a general hardware configuration. For this reason, this embodiment may be realized by a combination of an imaging device having an imaging function and a computer device having an image processing function. Also, if there is an image stack that has already been captured, it can also be realized by the computer device performing image processing using the image stack.

[0037] First, a multi-focus image stack ST is acquired by performing bright-field imaging multiple times with different focal depths on a raw sample S (embryo E in this example) that is a processing object (step S101). The image may be newly acquired by the imaging unit 13 performing imaging, or image data that has been previously captured and stored in an external storage or the like may be acquired from the outside via an appropriate recording medium or electric communication line.

[0038] Each image in the image stock thus obtained is converted into a heat map (step S102). Specifically, each of the multiple images included in the image stock is input to a machine learning model previously constructed by machine learning, and a heat map corresponding to each image is obtained as the output. As will be described in detail later, when the input image contains a clear image of a cell, the output of the machine learning model is a heat map in which a weight is assigned such that the position corresponding to the inside of the outline of the cell is larger than the outside of the cell and the closer to the center of the cell, the larger the weight is.

[0039] In each of the heat maps corresponding to each image, a peak point with a weight greater than that of the surroundings is detected (step S103). As can be seen from the characteristics of the heat maps described above, there is a high probability that the weight peak point represents the center position of each cell. Therefore, it can be said that there is a certain correlation between the number of peak points detected from each image and the number of cells that compose the cell cluster.

[0040] However, one cell may appear across multiple images with different focal depths, and in this case, the same cell may be detected as a peak point in multiple images. It is necessary to avoid counting these points in duplicate. For this purpose, peak points detected in multiple images that are likely to be derived from the same cell are consolidated into one (step S104). The method will be described later.

[0041] Each peak point after aggregation is considered to represent the center position of one cell. Therefore, the peak points are counted, and the counting result is regarded as the number of cells (step S105). This makes it possible to avoid counting a cell that appears in multiple images in duplicate, and enables automatic cell counting with high accuracy.

[0042] Next, a process for creating a machine learning model for outputting a heat map will be described. As described above, the heat map gives a greater weight to the clearer image of the cell contained in the original image, the closer to the center of the cell. On the other hand, if the image does not contain cells or the image is unclear, the weight is set to zero in the entire region.

[0043] In this way, the function of converting a cell image into a heat map in which the weighting is highest in the area that is most likely the center of the cell, and the weighting decreases the further away from the center the surrounding area is, can be achieved using a machine learning model constructed using supervised machine learning as follows.

[0044] FIG. 4 is a flowchart showing a process for constructing a machine learning model. FIG. 5 is a diagram illustrating a process for creating a heat map. First, images collected as teacher images are acquired (step S201). The teacher images are a plurality of images including images of cells captured in a focused state, particularly images with clear contours. For example, an expert can select an image in which cells appear clearly from a plurality of images obtained by multi-focus bright-field imaging of an embryo E, which is a processing object, or a sample S of the same kind, and use the selected image as the teacher image.

[0045] It is preferable to collect as many teacher images as possible, and also to deal with individual differences in cells, it is preferable to collect images of a plurality of samples S. In order to increase the number of teacher images and improve the learning accuracy, an image interpolated in the focal depth direction may be created using two images included in a multi-focus image stack that differ in focal depth by one step.

[0046] The image data of the teacher images collected in this manner is stored in, for example, storage 145. Then, one of the images is displayed on display unit 148 (step S202), and instruction input from the user for the displayed image is accepted (step S203). Note that multiple teacher images may be displayed on one screen.

[0047] The instruction input is performed to specify a clear cell region included in each teacher image. For example, as shown in FIG. 5(a), the user can perform the instruction input by tracing the clear outline of the cell C included in the display image Is using an appropriate drawing tool T. Alternatively, the instruction may be performed by superimposing a circle or ellipse that roughly matches the periphery of the cell C on the image. In this way, the user can specify the cell region in the image using an appropriate method, and each teacher image is labeled.

[0048] The teacher image may include an image that does not include a clear image of a cell, such as image Ia shown in FIG. 2(a). If the image does not include a cell or the image of the cell is unclear, the contour is not taught. This is technically equivalent to labeling the image as not having a clear image of a cell. Also, one teacher image may include multiple clear images of cells, such as image Ic shown in FIG. 2(a). In that case, it is desirable to teach the contour of each cell and to distinguish them so that they can be treated as independent.

[0049] A label image It is created using the region surrounded by the thus-instructed contour CT as a teacher label, and a heat map image Iu is created by weighting the inside of the contour (step S204). For all of the collected teacher images, instruction input is accepted (step S205).

[0050] As shown in Fig. 5(b), the heat map image Iu is an image in which a weight is given to the inside of the instructed contour CT so that the weight is larger than the outside and the weight is larger toward the center of the region surrounded by the contour CT. Note that the heat map image Iu shown here is expressed as an image with higher density, that is, lower brightness as the weight is larger, but conversely, it may be expressed as an image with higher brightness close to white as the weight is larger.

[0051] For example, the center of gravity of the area surrounded by the contour CT can be obtained, and weighting can be performed according to a two-dimensional Gaussian distribution centered on the center. Weighting can also be performed using a distance map that gives weights according to the distance from the contour CT to the inside of the area surrounded by the contour CT. In this way, it is possible to create a heat map image Iu by applying various weighting methods to the inside of the area surrounded by the contour CT, in which the weights are larger toward the center.

[0052] As shown in FIG. 5(b), the weight does not necessarily have to be zero at the position of the contour CT. In short, it is sufficient that the area surrounded by the contour CT can be distinguished from the outside area, and the weight is increased toward the center of the inside area. In this sense, the instruction of the contour CT does not have to strictly specify the contour of the cell in the image. Also, in order to more clearly distinguish the inside and outside of the cell, a certain offset value may be added to the weight given to the inside of the contour CT.

[0053] The magnitude of the weight given to each point on the heat map represents the relative probability that the point corresponds to the center of the cell. In other words, the position where the weight is maximum in the heat map represents the center position of the cell. The center of the cell here refers to the point farthest from other cells, etc., when the cell is in contact with them, and has no direct relationship to the structure within the cell, such as the position of the cell nucleus.

[0054] Next, machine learning is performed using a set of each of the multiple teacher images Is and the corresponding heat map image Iu as teacher data (step S206). As the machine learning algorithm, it is possible to appropriately select from various methods known as supervised machine learning methods and use them. For example, a convolutional neural network method can be used.

[0055] A machine learning model is constructed by executing machine learning using the teacher image Is as an input image and the corresponding heat map image Iu as a correct answer image. The machine learning model thus constructed has acquired the function of outputting a heat map representing the center position of a cell estimated from a newly given cell image using a probability distribution. In other words, the heat map image output from the trained machine learning model provides information for estimating the center position of a cell in the input image.

[0056] Specifically, if a peak point where the weighting is maximized compared to the surrounding area is searched for and detected in the output heat map image, the peak point indicates the position most likely to be the center of the cell. In an image that does not include a clear image of a cell, no substantial peak point appears. The detection of the peak point (step S103) can be performed, for example, as follows.

[0057] Fig. 6 is a diagram showing an example of a method for detecting peak points. Note that for ease of understanding, the relationship between the position and the weight in the heat map image is shown only in one dimensional direction, but the actual weight is distributed within a two-dimensional image plane.

[0058] The heat map image Ih output by the trained machine learning model, as shown in the upper part of the figure, is pre-processed to remove areas that do not need to be considered in subsequent processing. For example, areas where there are clearly no cells, such as the outside of the container, and areas that are significantly smaller than the expected cell size and are clearly not considered to be cells, are eliminated by binarization or masking. After that, a filter process is performed to remove fine irregularities caused by image noise, etc. For example, a moving average filter process can be applied. This results in a blurred image with a smooth probability distribution curve (actually a curved surface corresponding to a two-dimensional image), as shown in the curve A in the middle part of the figure. When maximum value filter process is performed on this blurred image, a maximum value image that bulges the probability distribution curve horizontally, as shown in curve B, is generated. There is no change in the value at the peak point with a heavier weight than the surroundings.

[0059] When the difference between the maximum value image and the blurred image is taken, the weighting difference ΔW is zero in the area far from the cell where the weighting is zero and at the peak point, as shown in the lower part of the figure. In this way, the peak point can be detected by detecting the point where the difference is zero. In order to distinguish between the area far from the cell and the peak point, an appropriate threshold value Wth is set for the weighting value as shown in the middle part of the figure, and the point where the weighting is larger than the threshold value Wth and the difference ΔW is zero is determined to be the peak point.

[0060] The peak points detected in this way can be considered to represent the center positions of cells that appear relatively clearly in each image in the multi-focus image stack. However, when one cell appears across multiple images, peak points representing the same cell will be detected in multiple images. In order to avoid duplicate counting caused by this, in this embodiment, peak points that are thought to originate from the same cell are associated with each other, and they are aggregated and considered as one peak (step S104).

[0061] FIG. 7 is a diagram explaining the process for aggregating peak points. In a cell mass formed by gathering cells of the same type, the cells are relatively uniform in size. In particular, embryos in a good state at the early development stage are formed by cells of a uniform size. For this reason, as shown in FIG. 7(a), it is considered that the center-to-center distance D between adjacent cells C, C will not be smaller than the minimum distance determined by the size of cell C. Even if there is variation in cell size, the same consideration can be applied by using the minimum distance derived from the size of relatively small cells.

[0062] The peak points detected from the heat map image Ih correspond to the central positions of the cells. Therefore, it can be said that the distance between the peak points detected from the multiple heat map images will not be smaller than the above-mentioned minimum distance if they are derived from different cells. For this reason, it is reasonable to consider that the detected peak points whose distance between them in real space is sufficiently smaller than a predetermined value assumed from the cell size are derived from the same cell. Based on this idea, the detected peak points are aggregated.

[0063] A more specific example will be described with reference to FIG. 7(b). An image in which only the peak points detected from the heat map image Ih are extracted and arranged on the map plane is called a peak image, and three peak images Ip1, Ip2, and Ip3 are obtained from three images captured with the focal depths differing by ΔV. Among these, it is assumed that peak points P1 and P2 are detected from the peak image Ip, peak points P3 and P4 are detected from the peak image Ip2, and peak points P5 and P6 are detected from the peak image Ip3. In order to clearly show the peak points on each peak image, the distance in the focal depth direction is enlarged in the figure compared to the distance in the direction along the map plane (hereinafter referred to as the "map surface direction").

[0064] Peak point P1 on peak image Ip1 and peak point P3 on peak image Ip2 are relatively close to each other in the map surface direction. The distance between these peak points in the focal depth direction is ΔV. Thus, peak points P1 and P3 on different peak images that are relatively close to each other in the map surface direction and the focal depth direction are highly likely to originate from the same cell. As will be described in detail later, the state of "close" here is a concept determined from the relationship with the expected size of the cell.

[0065] On the other hand, the peak points P1 and P2 on the peak image Ip1, the peak points P3 and P4 on the peak image Ip2, the peak points P2 on the peak image Ip1 and the peak points P3 and P4 on the peak image Ip2, and the peak points P1 on the peak image Ip1 and the peak points P4 on the peak image Ip2 are all spaced apart by a relatively large distance in the map surface direction. Therefore, these peak points are considered to originate from different cells.

[0066] In addition, the peak point P1 on the peak image Ip1 and the peak point P5 on the peak image Ip3 are relatively close to each other in the map surface direction, but the distance between them in the focal depth direction is 2ΔV, which is a relatively large distance. Therefore, these peak points P1 and P5 are considered to originate from different cells. Similarly, the peak point P4 on the peak image Ip2 and the peak point P6 on the peak image Ip3 are close to each other in both the map surface direction and the focal depth direction, and are considered to originate from the same cell.

[0067] In this way, peak points that are adjacent to each other in both the map surface direction and the focal depth direction, i.e., peak points P1 and P3, and peak points P4 and P6, are associated with each other as being derived from the same cell. On the other hand, peak points P2 and P5 that are not adjacent to other peak points in at least one of the map surface direction and the focal depth direction are regarded as isolated peak points derived from different cells.

[0068] Then, peak points that are mutually adjacent and associated are aggregated and treated as one group, while isolated peak points are treated as one group, and the number of groups after aggregation is counted (4 in this example). This makes it possible to obtain the number of cells while avoiding duplication caused by a single cell appearing in multiple images. In a machine learning model that has been constructed using appropriate teacher images and teachings for them, it is possible to automatically calculate the number of cells without the need for subjective human judgment.

[0069] Fig. 8 is a flow chart showing a first aspect of the adjacent peak aggregation process. Fig. 9 is a diagram showing an example of the adjacent peak aggregation process. Fig. 9(a) is a diagram showing the positions of peak points P(1) to P(10) in five peak images Ip1 to Ip5. Fig. 9(b) is a diagram showing these peak points arranged on a single map plane, which corresponds to an image in which the peak images Ip1 to Ip5 are superimposed.

[0070] First, an internal parameter k is set to 1 (step S301), and a search point P(k) is set (step S302). At this point, a peak point P(1) is set as the search point. Then, within the map plane shown in FIG. 9(b), a nearby peak point that is the shortest distance from the search point P(1) is searched for (step S303). In the example shown in FIG. 9(b), a peak point P5 that is closest to the search point P(1) within the map plane is detected as the nearby peak point.

[0071] When the distance between the search point P(k) and the detected nearby peak point in the map plane is equal to or less than the first threshold (YES in step S304) and the distance in the focal depth direction is equal to or less than the second threshold (YES in step S305), the nearby peak point is a peak point detected near the search point P(k) in real space, and it is reasonable to consider that they originate from the same cell. Therefore, this nearby peak point is associated with the search point P(k) by treating it as belonging to the cluster k that includes the search point P(k).

[0072] The first threshold value applied to the distance in the map surface direction and the second threshold value applied to the distance in the focal depth direction are values ​​that can be appropriately determined according to the size of the cells. Specifically, they can be values ​​corresponding to the minimum value assumed as the distance between the centers of adjacent cells. The map surface direction and the focal depth direction are expediently determined according to the situation during multi-focus imaging, and since cells do not inherently have anisotropy in these directions, there is no need in principle to set the first threshold value and the second threshold value to different values.

[0073] In reality, the resolution of the distance in the map plane can be reduced to, for example, about the pixel size, so the first threshold can be freely set according to the cell size. On the other hand, the resolution in the focal depth direction depends on the depth of field of the imaging optical system including the objective lens 131 and the pitch setting of the focal position adjustment during imaging. In addition, the actual focal position is affected by refraction due to the cells and culture medium. For these reasons, the second threshold is not necessarily determined in the same way as the first threshold. Provided that the depth of field of the imaging optical system is sufficiently small, the second threshold can be determined as follows, for example, based on the relationship between the pitch setting during imaging and the cell size.

[0074] For the purpose of multi-focus imaging, which focuses on each cell forming a cell cluster in sequence and captures the image, it is considered that the adjustment pitch of the focal position is set to be equal to or slightly smaller than the cell size. In this case, the range in which a cell clearly captured in one image can appear in other images can be considered to be adjacent images with a focal position setting that differs by only one step. In other words, the adjustment pitch of the focal position can be set as the second threshold. Similarly, for example, if the adjustment pitch of the focal position is about half the cell size, images with a focal depth that differs by two steps can be considered. In other words, the second threshold can be set to twice the adjustment pitch of the focal position.

[0075] Of course, it is technically equivalent to set the second threshold value in the same way as the first threshold value, and in step S305, calculate the distance between the peaks in the focal depth direction each time based on the adjustment pitch of the focal position and compare it with the second threshold value. However, as described above, it is easier and more realistic to predetermine how many steps of difference in focal depth are to be judged as "closeness."

[0076] In the example shown in Fig. 9(b), the second threshold is set to the adjustment pitch of the focal position. In this case, the peak P(5) detected as the closest peak point to the search point P(1) exceeds the second threshold in the focal depth direction because the focal depths of the peak images Ip1 and Ip3 in which they were detected differ by two steps when they were captured. Therefore, no association is made between the search point P(1) and the closest peak point P(5).

[0077] If the result of step S304 is NO, that is, if the detected nearby peak point is at a distance greater than the first threshold value from the search point P(k), there is no other peak point that may be associated with the search point P(k). Therefore, the internal parameter k is incremented by one, and the process returns to step S302 (step S312), and the same process is performed with another peak point, for example P(2), as a new search point. If the peak point is already associated with another peak point, it may be excluded from the search. If the search for all peak points has been completed, the process is terminated (step S311).

[0078] If the result of step S305 is NO, that is, if the distance between the detected adjacent peak point and the search point P(k) is equal to or less than the first threshold in the map plane but exceeds the second threshold in the focal depth direction, another peak point that is next closest to the search point P(k) after the current adjacent peak point in the map plane is searched for (step S303). The processes in and after step S304 are also performed on the peak point thus detected.

[0079] In this way, even if there are multiple peak points in the vicinity of the search point P(k) in the map plane, it is possible to evaluate all of them. In Fig. 9(b), the dotted lines represent circles with each peak point as the center and a radius of the first threshold value. When one of the peak points P(k) is set as a search point, other peak points within the circle with the peak point as the center are sequentially searched for, and the distance between them in the focal depth direction is evaluated, thereby proceeding with peak aggregation.

[0080] In the example shown in FIG. 9(b), the peak point closest to the search point P(1) after the peak point P(5) is the peak point P(3), and the focal position setting at the time of imaging is one step different between the peak image Ip1 including the search point P(1) and the peak image Ip2 including the peak point P(3). Therefore, the distance in the focal depth direction between the search point P(1) and the other peak point P(3) is judged to be equal to or less than the second threshold, and the two are associated as originating from the same cell. When counting, the two peak points are counted as one cell.

[0081] Similarly, by evaluating the distance between each peak point in the map surface direction and the focal depth direction, peak point P(4) and peak point P(6) are associated, and peak point P(7) and peak point P(9) are associated, and each of these is counted as one cell. On the other hand, isolated peak points P(2), P(5), P(8), and P(10), which are not associated with other peak points, are each counted as one cell. Therefore, in the example shown in FIG. 9(b), although the number of peak points is 10, those thought to be derived from the same cell are aggregated, and the cell count result is 7. In this way, it is possible to avoid counting one cell more than once.

[0082] The number of peak points detected from each image in the multi-focus image stack varies, and for example, one image may not contain any peak points. In addition, there may be a situation where there are no peak points that are related to each other among the detected multiple peak points. The above method can handle such cases and can count the number of cells with high accuracy.

[0083] Fig. 10 is a flow chart showing a second aspect of the adjacent peak aggregating process. Fig. 11 is a diagram showing an example of this adjacent peak aggregating process. Compared to the first aspect shown in Fig. 8, it is different in that step S307 is provided after step S306. Since the processing contents are otherwise the same, the same numbers are used for processing steps having the same processing contents, and the description thereof will be omitted.

[0084] In this embodiment, when the peak points are associated with each other in step S306, the center of the search point P(k) is changed and set to the center position of the cluster formed by the associated peak points. The "x" mark in Fig. 11 represents the cluster center. Then, the search for nearby peak points in the map plane thereafter is performed based on a comparison between the distance from the cluster center and the first threshold value. The dotted line in Fig. 11 represents a circle centered on the cluster center and with a radius equal to the first threshold value.

[0085] The peak points identified by the above method from each image obtained by multi-focus imaging indicate the center position of the cell image in the image, but the detected position does not necessarily coincide with the center of the cell in real space. By using the center position of a cluster composed of mutually associated peak points as the search point, it is possible to perform the search from a position closer to the center of the cell in real space. Similarly, the distance may be evaluated based on the center position of the cluster in the focal depth direction.

[0086] In the first embodiment shown in Fig. 8, by setting the position of each peak point as the center of the search, adjacent peak points may be linked together in a chain reaction to form a large cluster, especially when cells are densely packed. In that case, a cluster whose size is much larger than the size of a cell may be regarded as a single cell. By setting the center position of the cluster as the search point, it is possible to avoid such a problem in advance.

[0087] As a practical matter, if the focal position adjustment pitch, the first threshold value, and the second threshold value are each appropriately set in accordance with the expected cell size, the above-mentioned problems will not arise and there will be no significant difference in the results between the two processing modes.

[0088] As described above, in this embodiment, each of the multiple images obtained by multi-focus imaging of a cell cluster is converted into a heat map in which the weight increases as the image approaches the center of the cell, using a machine learning model that has been trained in advance. Then, peak points where the weight is maximized in the heat map are detected, and the distance between the peak points is evaluated in each of the map surface direction and the focal depth direction, and peak points that are highly likely to originate from the same cell are associated with each other and aggregated. The distance is evaluated by comparing with a threshold value determined from the expected cell size. Therefore, it is possible to avoid counting multiple peak points originating from the same cell in duplicate, and it is possible to accurately count the number of cells that constitute the cell cluster by counting the number of peaks.

[0089] The machine learning model is constructed by supervised learning based on a teacher image containing a clear image of a cell and a ground truth image in which a heat map is arranged inside the contour taught in the teacher image. Therefore, the center position of the cell in the image can be accurately identified. Then, by performing an evaluation based on the distance between the cell centers identified from the cell contour, it is possible to distinguish peak points originating from the same cell from peak points originating from different cells without being influenced by the texture inside the cell as in the conventional technology that uses, for example, stained cell nuclei.

[0090] The present invention is not limited to the above-mentioned embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, the imaging device 1 of the above-mentioned embodiment has an imaging function for acquiring a multi-focus image stack by itself, and an image processing function for analyzing the images and performing cell counting processing. However, the cell counting method according to the present invention can also be executed by a computer device that does not have an imaging function itself, but acquires an image stack obtained by imaging with another device that has an imaging function. To make this possible, the present invention may be implemented as a software program for causing a computer device to execute each processing step of the above-mentioned processing.

[0091] Such a program can be distributed, for example, by downloading it via a telecommunications line such as the Internet, or by distributing a computer-readable recording medium in which the program is recorded. In addition, for example, by reading the program into an existing microscope imaging device via an interface, the present invention can be implemented by the device.

[0092] In the above embodiment, the construction of the machine learning model based on the collected teacher data and the cell counting process using the constructed machine learning model are performed in the same device, but these may be performed in different devices. For example, if a machine learning model is constructed using a computer device with high computing power and the result is implemented in an imaging device, even an imaging device with limited computing power can be responsible for imaging and cell counting.

[0093] In the peak aggregation process of the above embodiment, the distance between peaks is evaluated separately in the map surface direction and the focal depth direction, but instead, the distance between peaks may be evaluated three-dimensionally based on whether or not the peak is within a sphere of a predetermined radius centered on the search point. For example, the peaks can be aggregated by applying a clustering method such as the X-means method.

[0094] In addition, in the above embodiment, an embryo (fertilized egg) in an early stage of development is used as the cell mass to be processed in the present invention, but the cell mass to be processed in the present invention is not limited to this. In particular, the present invention can be suitably applied to the purpose of automatically counting the number of cells constituting a cell mass consisting of a plurality of cells of uniform size.

[0095] As described above by way of example and with reference to specific embodiments, in the cell counting method according to the present invention, among a plurality of peaks detected from different heat maps and whose distance from each other in a direction along the map plane is smaller than a first threshold value, those peaks whose difference in focal depth when the images corresponding to the heat maps in which the peaks were detected were captured are smaller than a second threshold value can be associated with each other. Here, the first threshold value and the second threshold value may be the same value.

[0096] Moreover, it is preferable that the teacher image is an image of a cell cluster of the same type as the cell cluster to be counted. In this way, the characteristics of the cells contained in the cell cluster to be counted can be effectively reflected in the learning result, and cells can be extracted from the image with high accuracy.

[0097] In this case, the training image preferably includes an image focused on the periphery of at least one cell. By performing machine learning using an image with a clear cell outline as a training image, it is possible to generate a heat map that appropriately indicates the center position of the cell for images with similarly clear outlines in the image stack.

[0098] Furthermore, the teacher image may be selected from a plurality of images of the cell cluster captured in bright field at different focal depths. Each of the images of the cell cluster containing a large number of cells captured at various focal depths may contain clear images of cells at positions corresponding to the focal depths at which the images were captured. By using images containing images of such cells as teacher images, the shape, color, and the like of the cells in the cell cluster can be effectively reflected in the learning results.

[0099] In addition, in the method for constructing a machine learning model for cell counting according to the present invention, for example, for each acquired teacher image, a teaching input for teaching the contour of a cell in the teacher image is received, and in the heat map, a weight is given to the inside of the region surrounded by the taught contour that is larger than that of the outside and the closer to the center of the region, so that a machine learning model that can accurately indicate the center position of a cell without being affected by the texture inside the cell can be constructed by first identifying the contour of the cell and then giving an appropriate weight distribution to the inside of the cell.

[0100] The machine learning model only needs to have the ability to estimate the center of a cell with sufficient accuracy to evaluate the distance to other cells. For this reason, the weighting in the heat map can be any distribution that is larger inside the cell than outside and larger closer to the center of the cell. A representative method is, for example, a two-dimensional Gaussian distribution centered on the cell center. [Industrial Applicability]

[0101] This invention makes it possible to automatically and accurately count the number of cells in a cell mass consisting of multiple cells, and can effectively assist in the evaluation of the condition of cultured embryos (fertilized eggs) in the field of assisted reproductive technology, for example. [Explanation of symbols]

[0102] 1. Imaging device 10 Sample container 13 Imaging unit 14 Control section C cell E Embryo (cell mass) Ih,Iu Heatmap Is Teacher Image S sample ST image stack

Claims

1. A cell counting method for counting the number of cells constituting a cell cluster, comprising the steps of: acquiring an image stack including a plurality of images of the cell mass captured in bright field at different focal depths; generating a plurality of heat maps corresponding to each of the plurality of images using a machine learning model; detecting peaks from each of the heat maps; correlating the detected peaks with each other, the peaks belonging to different heat maps and the peaks having a distance along the map plane and a distance in the focal depth direction that are each smaller than a predetermined value; counting the number of said peaks by regarding the associated peaks as one peak; Equipped with The machine learning model is A teacher image, which is a bright-field image of a cell; A correct answer image is a heat map in which the inside of the contour of the focused cell in the teacher image is weighted larger than the outside of the contour and the closer it is to the center of the cell, the greater the weight is. The cell counting method is constructed by machine learning using a set of training data.

2. The cell counting method of claim 1, wherein among a plurality of peaks detected from different heat maps and whose distance from each other in a direction along the map plane is less than a first threshold, those peaks whose difference in focal depth when the image corresponding to the heat map in which the peak was detected was captured is less than a second threshold are associated with each other.

3. The cell counting method according to claim 2 , wherein the first threshold value and the second threshold value are equal to each other.

4. 4. The cell counting method according to claim 1, wherein the teacher image is an image of a cell cluster of the same type as the cell cluster to be counted.

5. A program for causing a computer to execute each step of the cell counting method according to any one of claims 1 to 4.

6. acquiring a plurality of teacher images, which are bright-field images of cells; A step of creating a ground truth image, which is a pseudo heat map, for each of the teacher images, in which a weight is given to the inside of the contour of a cell that is in focus in the teacher image, which is greater than the outside of the contour, and which is greater the closer it is to the cell center, while no weight is given to an image of a cell that is not clear; A step of executing machine learning using a set of the teacher image and the correct image as teacher data to construct a machine learning model; A method for constructing a machine learning model for cell counting, comprising:

7. The method for constructing a machine learning model for cell counting according to claim 6 , wherein the training image includes an image focused on the periphery of at least one cell.

8. The method for constructing a machine learning model for cell counting according to claim 6 or 7, wherein the training image is selected from a plurality of images of a cell cluster captured in bright field at different focal depths.

9. For each of the acquired teacher images, a teaching input is received indicating a contour of a cell in the teacher image; A method for constructing a machine learning model for cell counting as described in any one of claims 6 to 8, wherein the heat map assigns a weight to the inside of the area surrounded by the taught contour that is greater than the outside of the contour and that is greater the closer it is to the center of the area.

10. The method for constructing a machine learning model for cell counting according to any one of claims 6 to 9, wherein the weighting in the heat map follows a two-dimensional Gaussian distribution centered on the cell center.

11. A program for causing a computer to execute each step of the method for constructing a machine learning model for cell counting according to any one of claims 6 to 10.

12. A computer-readable recording medium on which the program according to claim 5 or 11 is non-transitoryly recorded.

Images