Cell image analysis method

Abstract

The invention relates to a cell image analysis method. In order to provide a highly accurate cell image analysis method, provided is a cell image analysis method comprising: an image acquisition step for acquiring a time-series cell image group, the time-series cell image group is obtained by collecting a plurality of cell images obtained at a plurality of consecutive different timings in association with the timings; a region extraction step for extracting a cell candidate region from the cell image; a region tracking step of collecting cell candidate regions determined to correspond to the same subject in association with timing for cell candidate regions on the plurality of cell images included in the time-series cell image group, and acquiring the determined cell candidate regions as a time-series cell candidate region group; and an analysis step of analyzing information on the state of the cell, in which the analysis step includes using the trained model.

Description

Technical Field

[0001] The present disclosure relates to a cell image analysis method, a program, a medium, a cell image analysis device, and a cell image analysis system. Background Art

[0002] In the field of cell culture, cell images are often analyzed to analytically determine the state of cells, such as whether the cells are alive or dead. In particular, technology that analyzes cell images using machine learning is attracting attention.

[0003] Japanese Patent Application Laid-Open No. 2017-085966 discloses a technique for non-invasively identifying live and dead cells from bright field images. This technique identifies live and dead cells based on fluorescence images to obtain labeled data for machine learning.

[0004] Japanese Patent Application Laid-Open No. 2015-210212 discloses a technique for distinguishing whether cells are alive or dead based on morphological features of time-series cell images. This technique also describes using stained samples as labeled data for machine learning.

[0005] However, in the related art, there is still room for improvement in distinguishing whether cells are alive or dead. Summary of the Invention

[0006] An object of the present invention is to provide a cell image analysis method by which highly reliable labeling data can be created.

[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present disclosure, a cell image analysis method is provided, comprising: an image acquisition step, wherein the image acquisition step acquires a time series cell image group, wherein the time series cell image group is obtained by collecting a plurality of cell images obtained by any one of a bright field observation method and a phase contrast observation method at a plurality of consecutive different timings in association with the plurality of consecutive different timings; a region extraction step, wherein the region extraction step extracts a cell candidate region from the plurality of cell images included in the time series cell image group; and a region tracking step, wherein the region tracking step determines the cell candidate region on the plurality of cell images associated with mutually different timings included in the time series cell image group. Whether the candidate areas correspond to the same object, collecting the cell candidate areas determined to correspond to the same object in association with timing, and obtaining the determined cell candidate areas as a time series cell candidate area group; and an analysis step, which analyzes information about the state of the cell based on the information about the time series cell candidate area group, wherein the analysis step includes using a trained model, and wherein the trained model is a machine learning model, which is based on the information about the time series cell candidate area group and the information about the state of the cell obtained from the cell used for training, by using the information about the time series cell candidate area group as input and using the information about the state of the cell as output.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a processing flow chart of the cell image analysis method according to the first embodiment.

[0010] Figure 2 is a diagram for illustrating an example of the time-series phase contrast image group acquired in step S101 .

[0011] Figure 3 is a diagram for illustrating an example of the mask image acquired in step S102 .

[0012] Figure 4 is a diagram for illustrating an example of the area tracking result acquired in step S103 .

[0013] Figure 5 : is a table showing an example of the feature value data group calculated in step S104 and step S105 .

[0014] Figure 6 It is a processing flow chart of the trained model generation method in the first embodiment.

[0015] Figure 7is a diagram for illustrating an example of a time-series phase contrast image group and a fluorescence image acquired in step S601 .

[0016] Figure 8 ] is a table showing an example of the feature value data group calculated in steps S604 to S606.

[0017] Figure 9 1 is a diagram showing an example of a time transition graph of fluorescence intensity for each time series cell candidate region group.

[0018] Figure 10 _ is a table showing an example of the fluorescence intensity feature value data set of the corresponding time-series cell candidate region group calculated in step S606 .

[0019] Figure 11 is a table showing examples of fluorescence intensity feature value data groups of respective time-series cell candidate region groups subjected to labeling processing.

[0020] Figure 12 is a table showing an example of a feature value data set for each cell candidate region subjected to labeling processing.

[0021] Figure 13 is a set of graphs showing examples of time transition graphs of the fluorescence intensity of each time series cell candidate region group for each label.

[0022] Figure 14 is a table showing an example of obtaining a binary value for each label by the trained model in step S106 .

[0023] Figure 15 is a table showing an example of obtaining a likelihood value for each label by the trained model in step S106 .

[0024] Figure 16 : is a diagram for illustrating an example of a time-series phase contrast image group and a fluorescence image acquired in the second modified example of the first embodiment.

[0025] Figure 17 is a data flow diagram of the step-by-step labeling process in the fourth modified example of the first embodiment.

[0026] Figure 18 is a set of graphs for illustrating an example of extracting data of live cells and dead cells from data of cells marked as having a change in life-death state in the fifth modified example of the first embodiment.

[0027] Figure 19 is a set of graphs for illustrating an example of the result of performing cleaning processing on data labeled as non-cells in the sixth modified example of the first embodiment.

[0028] Figure 20 is a processing flow chart of the cell image analysis method according to the second embodiment.

[0029] Figure 21A is a part of a diagram for describing input and output of a trained model in the second embodiment, and is a table for showing an example of a feature value data group.

[0030] Figure 21B is a part of a diagram for describing input and output of a trained model in the second embodiment, and is a table for showing an example of a result of extracting time series feature value data.

[0031] Figure 21C is a part of a diagram for describing input and output of a trained model in the second embodiment, and is a table for showing an example of a result in which a binary value is output.

[0032] Figure 22 4 is a processing flow chart of the trained model generation method in the second embodiment.

[0033] Figure 23 is a processing flow chart of the cell image analysis method according to the third embodiment.

[0034] Figure 24A It is a part of a diagram for illustrating an example of input data of a trained model in the third embodiment, and is a diagram for illustrating an example of a partial cell image group cut out from a corresponding cell image based on a region tracking number and a cell candidate region.

[0035] Figure 24B is a part of a diagram for describing input and output of a trained model in the third embodiment, and is a table for showing an example of a result in which a binary value is output.

[0036] Figure 25 4 is a processing flow chart of the trained model generation method in the third embodiment.

[0037] Figure 26 is a block diagram for illustrating a hardware configuration example of an information processing device capable of executing a program according to the present disclosure.

[0038] Figure 27 : is a diagram for illustrating an example of a time-series phase contrast image group and a fluorescence image acquired in the third modified example of the first embodiment.

[0039] Figure 28 is a diagram for illustrating an example of a cell image analysis device.

[0040] Figure 29 is a diagram for illustrating an example of a cell image analysis system.

[0041] Figure 30 is a table showing examples of evaluation results of the living cell determination model.

[0042] Figure 31 It is a graph showing the results of evaluating the accuracy (viable cells) per hour of culture time.

[0043] Figure 32 It is a graph showing the results of evaluating the accuracy (non-viable cells) for each culture time.

[0044] Figure 33 : is a table showing the results of calculating the evaluation index for each of the culture time slot (0 to 11 hours) and the culture time slot (12 hours or longer).

[0045] Figure 34 It is a graph showing the results of evaluating the accuracy (viable cells) per hour of culture time.

[0046] Figure 35 It is a graph for showing the results of evaluating the accuracy (non-viable cells) per hour of culture time. DETAILED DESCRIPTION

[0047] Now, embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below do not limit the invention defined in the claims. A number of features are described in the embodiments, but the present invention does not necessarily require all of these features, and multiple features may be appropriately combined. In addition, in the drawings, the same or similar components are represented by the same reference numerals, and their repeated descriptions are omitted.

[0048] (Problems to be Solved by the Cell Image Analysis Method According to This Embodiment)

[0049] The state of a cell (such as whether the cell is alive or dead) changes over time. For this reason, training data created based only on fluorescence images at a specific time (such as in Japanese Patent Application Laid-Open No. 2017-085966 and Japanese Patent Application Laid-Open No. 2015-210212) makes it possible to train a machine learning model by using data to which inaccurate basic fact labels are assigned. Analysis based on such a machine learning model may lack performance. According to the present disclosure, more reliable labeled data can be created. By using labeled data created according to the present disclosure, the performance of a machine learning model for distinguishing cell states can be improved. A specific description thereof is as follows.

[0050] According to the present disclosure, a cell image analysis method is provided, including: an image acquisition step of acquiring a time-series cell image group, the time-series cell image group being acquired by collecting a plurality of cell images acquired at a plurality of consecutive different timings by either a bright field observation method or a phase contrast observation method in association with the plurality of consecutive different timings; a region extraction step of extracting cell candidate regions from the plurality of cell images included in the time-series cell image group; a region tracking step of determining whether the cell candidate regions on the plurality of cell images associated with mutually different timings included in the time-series cell image group correspond to the same object, collecting the cell candidate regions determined to correspond to the same object in association with the timings, and acquiring the determined cell candidate regions as a time-series cell candidate region group; and an analysis step of analyzing information about the state of the cell based on information about the time-series cell candidate region group, wherein the analysis step includes using a trained model, and wherein the trained model is a machine learning model trained based on information about the time-series cell candidate region group and information about the state of the cell acquired from cells used for training, by using the information about the time-series cell candidate region group as input and using the information about the state of the cell as output.

[0051] The trained model may be configured to acquire information about the state of the cell based on the fluorescence intensity feature value data, the fluorescence intensity feature value data including at least information about a temporal variation in fluorescence intensity.

[0052] In a cell image analysis method, a trained model can be obtained by performing the following operations on cells used for training: a fluorescence image acquisition step, which acquires a time series fluorescence image group, which is obtained by collecting fluorescence images corresponding to multiple cell images included in the time series cell image group in association with timing; a fluorescence intensity feature value calculation step, which acquires fluorescence intensity feature value data of a time series cell candidate area group based on the time series fluorescence image group; a labeling step, which assigns a label related to the state of the cell to the cell candidate area based on the fluorescence intensity feature value data; and a trained model generation step, which generates a trained model by using information about the state of the cell based on the label related to the state of the cell as output and information about the time series cell candidate area group as input for training, and the fluorescence intensity feature value data may include information about the temporal change of at least the fluorescence intensity.

[0053] The information about the state of the cell may include at least information about a scalar value indicating one of binary values ​​or indicating a possibility of whether the cell candidate region is a live cell region.

[0054] In addition, the information about the state of the cell may include at least information about a scalar value indicating one of binary values ​​or indicating the possibility of whether the cell candidate region is a differentiated region.

[0055] [First embodiment]

[0056] (Overview)

[0057] In the first embodiment, a method for obtaining information about the state of a cell by inputting a eigenvalue vector formed by intensity eigenvalues ​​and morphological eigenvalues ​​or a eigenvalue vector formed by eigenvalues ​​related to time changes in intensity eigenvalues ​​and morphological eigenvalues ​​into a trained model, and a method for generating a trained model are described, wherein the intensity eigenvalues ​​and morphological eigenvalues ​​are calculated based on a time series cell image group (i.e., a time series phase contrast image group) obtained by collecting multiple cell images taken by a phase contrast observation method in association with timing.

[0058] As an exemplary application of this embodiment, an example is described in which a cell sample is observed immediately after inoculation of stem cells (induced pluripotent stem cells (iPS cells) or embryonic stem cells (ES cells)) and information indicating whether each cell is a "live cell," "dead cell," or "non-cell" is obtained as information about the state of the cell. The information about the state of the cell is not limited to this and may be information indicating whether each cell is a "live cell" or a "dead cell," or information indicating whether each cell is "differentiated" or "pre-differentiated."

[0059] (Processing process)

[0060] Figure 1 is a flowchart of the cell image analysis method according to the first embodiment.

[0061] (Step S101: Acquisition of a Time-Series Cell Image Group)

[0062] Step S101 is an image acquisition step in which a cell culture observation apparatus acquires a plurality of cell images captured by a bright field observation method or a phase contrast observation method at a plurality of consecutive different timings in association with the timings.

[0063] In step S101, cell image data captured by a cell culture observation device is acquired. Here, the cell image data is a time series of phase contrast images captured at predetermined intervals over a certain period of time relative to a single culture container. For example, the time immediately after inoculation is set to 0 hours, and 52 images are acquired at one-hour intervals until 51 hours later. Figure 2 1 is a diagram for illustrating an example of acquiring a time-series cell image group 202 with respect to the culture container 201. Cell images I00 to I51 correspond to images captured at corresponding times from 0 hour to 51 hour of culture time, respectively.

[0064] The image to be acquired in this step is not limited to a phase contrast image and may be any image as long as the image has a feature that emphasizes the contrast of the outline of cells relative to the non-cell area. For example, this step can be appropriately applied to images such as images picked up in a differential interference optical system, an oblique illumination system, or an optical system in which the object side and the image side are telecentric (a double-sided telecentric optical system), as well as defocused images (bright field images captured by shifting a fixed distance from the focused position in the optical axis direction).

[0065] In addition, each cell image is preferably an image captured so as to include a common field of view relative to the culture container. In this embodiment, it is assumed that all cell images I00 to I51 are captured with the same field of view. Cell image data can be acquired in real time from a cell culture observation device or from an external storage area (such as an HDD or cloud storage device).

[0066] This step acquires a time-series cell image group indicating the form of each cell captured by the cell culture observation device, and outputs the data to the module for executing step S102 (e.g., region extraction module 2802). The term "outputting information" includes outputting information after storing it in a storage device.

[0067] (Step S102: Acquisition of Cell Candidate Regions)

[0068] Step S102 is a cell candidate region acquisition step for acquiring a cell candidate region in each cell image included in the time-series cell image set acquired in step S101. As used herein, a "cell candidate region" refers to an object region whose outline is emphasized in the phase contrast image acquired in step S101. Specifically, this includes not only cell regions but also regions of objects such as cell debris and scratches or adhering matter on the culture container. In step S102, these regions are acquired as cell candidate regions.

[0069] Cell candidate regions in the phase contrast image can be obtained using well-known image analysis techniques. For example, image processing such as the active contour method or GraphCut can be used, or a trained model such as a deep learning network such as U-Net can be trained using a mask image indicating each cell candidate region as a teacher.

[0070] In this embodiment, an example is described in which a cell candidate region is acquired by a difference filter and through binarization processing.

[0071] First, a difference image is generated by applying a difference filter to the cell image. This is achieved by calculating the change in intensity between each pixel and its surrounding pixels and expressing the calculated change as an image. In the case of a cell image, the difference image is an image with high intensity values ​​in the outline of the cell candidate region and in the outlines of the cells within the cell candidate region.

[0072] Subsequently, the area with high intensity values ​​in the differential image is identified by performing a binarization process on the generated differential image. In the binarization process, any threshold value is set, and when the value of each pixel of the differential image is greater than or equal to the threshold value, the value is replaced by 1, and when the value is less than the threshold value, the value is replaced by 0. How the binarization process is performed is not limited to the method in which any threshold value is set. For example, a method of automatically determining the threshold value, such as Otsu binarization or binarization by the Li algorithm, can be used. In the case of setting any threshold value, the threshold value is set to be suitable for the shooting conditions of the device, such as the length of exposure and the focus setting. A method of determining a threshold for each pixel of the image, such as adaptive binarization, can also be used. Through the binarization process, a binary image (hereinafter referred to as an "edge image") is created, which is expressed by setting the pixel value in the area where the change in intensity value is large to 1 and the pixel value in other areas to 0.

[0073] Then, a mask image of the cell candidate region is generated based on the generated edge image. Here, the mask image is a binary image in which the cell candidate region is represented by a pixel value of 1, while other regions are represented by a pixel value of 0. The mask image is generated by extracting regions in the edge image that are linked to a pixel value of 1 and replacing the pixel value in each linked region with 1.

[0074] In this case, when the size of the target cell can be estimated, in this step, a range such as an upper limit and a lower limit can be set for the size of the cell candidate region, and the cell candidate region outside the range can be excluded. For example, in the case of stem cells, the exclusion area is set to 10000 μm. 2 The above areas.

[0075] By performing the above-mentioned processing, the cell candidate region in each image acquired in step S101 is acquired.

[0076] Figure 3 is used to illustrate the application of the above-mentioned processing to Figure 2 The corresponding cell images I00 to I51 are obtained from the mask images M00 to M51. The area of ​​black pixels in each mask image indicates the corresponding cell candidate area. Figure 3 As shown in , identifiers such as “C001” and “C002” are assigned thereto so that it is possible to distinguish to which cell image each cell candidate region belongs.

[0077] Through this step, as information indicating the cell candidate region in each cell image in the time series cell image group acquired in step S101, a mask image in which each cell candidate region can be identified and contour coordinate information corresponding to each cell candidate region are acquired, and the acquired information is output to a module for executing step S103 (e.g., region tracking module 2803).

[0078] (Step S103: Tracking of Cell Candidate Regions)

[0079] Step S103 is a region tracking step of determining whether the cell candidate regions on multiple cell images associated with different timings obtained in step S102 are the same region (corresponding to the same object), collecting the cell candidate regions determined to correspond to the same object in association with the timing, and obtaining the determined cell candidate regions as a time series cell candidate region group.

[0080] For cell candidate regions acquired in a cell image acquired at a specific timing (i.e., associated with a specific timing), it is possible to determine which cell candidate region is the same cell candidate region at the next timing by calculating the degree of overlap between the regions. For example, the degree of overlap "m" between two regions A1 and A2 is calculated using Expression 1.

[0081] m=S(A1∩A2) / MIN(S(A1),S(A2))...Expression 1

[0082] In Expression 1, S(A1) and S(A2) represent the areas of regions A1 and A2, respectively. Furthermore, S(A1∩A2) represents the area of ​​a region where region A1 and region A2 overlap with each other.

[0083] Now, the following describes the Figure 4 An example of determining which cell candidate regions on consecutive cell images are the same region.

[0084] First, the degree of overlap "m" between the cell candidate region C001 in the cell image M00 and each cell candidate region in the cell image M01 is calculated, and the cell candidate region with the largest degree of overlap in the cell image M01 and the cell candidate region C001 are considered to correspond to the same object. In this case, these cell candidate regions are said to be the same. Figure 4 In the example shown in FIG, the cell candidate region C001 and the cell candidate region C011 are determined to be the same region.

[0085] Subsequently, for the next cell candidate region C002, the degree of overlap "m" with each cell candidate region in the cell image I01 is calculated in the same manner as for the cell candidate region C001, and the cell candidate region with the largest degree of overlap is determined to be the same cell candidate region as the cell candidate region C002. At this time, the cell candidate region C011, which has been determined to be the same as the cell candidate region C001, is excluded from the object.

[0086] Thereafter, the same process is repeatedly performed on other cell candidate regions and other cell images, thereby determining which cell candidate regions are the same region on cell images associated with different timings.

[0087] In cell culture, cell candidate regions also include regions corresponding to detached, adherent cells and floating debris. In this case, a cell candidate region identical to a cell candidate region at one timing may not exist at the next timing. Therefore, it is effective to set a threshold value for the degree of overlap and determine that the cell candidate regions are identical when the degree of overlap with the cell candidate region with the maximum degree of overlap is equal to or greater than the threshold value. The threshold value is set to, for example, "0.25."

[0088] Furthermore, it is assumed that the regions determined to be the same in this step are assigned such identifiers that it is possible to distinguish which cell candidate regions are the same region. Figure 4 This is used to illustrate the application of this step to Figure 3 FIG 2 is a diagram showing examples of some of the identifiers assigned based on the results of the cell candidate regions. The cell candidate regions C001 and C011 and the cell candidate regions C121 and C511, as well as the cell candidate regions C002 and C012 and the cell candidate regions C122 and C512 are determined to be the same region, and identifiers T001 and T002 are assigned thereto. Each identifier (such as Figure 4 ) is hereinafter referred to as a “region tracking number.” Furthermore, a group of cell candidate regions associated with different timings to which the same region tracking number is assigned is referred to as a “time-series cell candidate region group.”

[0089] As described above, through this step, for the cell candidate regions obtained in step S102, region tracking numbers indicating which cell candidate regions are the same cell candidate region at different timings are assigned to the mask image and the contour coordinate information corresponding to each cell candidate region. This information is output to the module for executing step S104.

[0090] As a suitable example, an example has been described in which the degree of overlap is calculated based on Expression 1 to determine whether the regions are identical. However, without being limited to Expression 1, determination can be performed using, for example, a well-known evaluation index related to the degree of overlap between regions (such as DICE or IoU) or the movement distance of the center of mass of the region. Well-known region tracking methods such as methods for calculating optical flow or a Kalman filter can also be applied.

[0091] (Step S104: Calculation of the First Eigenvalue)

[0092] Step S104 is a first feature value calculation step of calculating the intensity feature or the morphological feature or both of each cell candidate region included in each time series cell candidate region group acquired up to step S103 .

[0093] The intensity feature value is independent of temporal variation and is a scalar value obtained by calculating statistics of the intensity values ​​of the cell candidate region, and examples of the statistics include the mean, minimum, and deviation. The morphological feature is a scalar value obtained by quantifying the size or shape of the cell candidate region, and, for example, the area or diameter is calculated. The intensity feature value and the morphological feature value are calculated for each cell candidate region acquired up to step S103.

[0094] The calculated feature value is linked to the identification number of the cell candidate region so that it is possible to distinguish to which cell candidate region the feature value belongs.

[0095] Through this step, the data set of the intensity feature value and the morphological feature value of the corresponding cell candidate region obtained up to step S103 is calculated and obtained, and the information is output to the module for executing step S105.

[0096] The calculated feature values ​​are merely examples. For example, the maximum value or median value may be calculated as the intensity feature value, and the roundness or firmness may be calculated as the morphology feature value.

[0097] (Step S105: Calculation of the Second Eigenvalue)

[0098] Step S105 is a second feature value calculation step of calculating a feature value (time-series feature value) indicating a temporal change of the first feature value or position information acquired in step S104 based on each time-series cell candidate region group acquired in step S103 .

[0099] For example, the second feature value is obtained by extracting a data set of the first feature value or position information of the same cell candidate region at a fixed time period in the past, and calculating the average value of the difference between the change amount based on linear approximation and the difference between each piece of data that is continuous in time. For example, based on each piece of data in the past 12 hours, the change amount of the diameter based on linear approximation and the average movement amount of the center of gravity coordinate are calculated. Specifically, the description of the first feature value or position information of the same cell candidate region at a fixed time period in the past is as follows: Figure 4 An example of calculating the amount of change in diameter and the average movement of the center of mass of the cell candidate region C121 in the example.

[0100] First, for the cell candidate region C121, the diameter data and centroid coordinates of the same cell candidate regions C001 to C011 in the past 12 hours are obtained. Subsequently, the change in diameter is calculated based on the diameter data obtained for 12 hours. The change in diameter can be obtained by treating the diameter data for 12 hours as a data set formed by two variables, time and diameter, and calculating the slope when performing linear approximation by the least squares method. In addition, the average movement amount is calculated based on the centroid coordinate data obtained for 12 hours. The average movement amount can be obtained by calculating the Euclidean distance between each centroid coordinate data that is continuous in time and calculating its average value. This processing as described above is also repeated for other time series cell candidate region groups, thereby obtaining the second eigenvalue data group for each time series cell candidate region group.

[0101] Figure 5 Table 500 is calculated Figure 4 An example of the results of the first eigenvalue data and the second eigenvalue data in each cell candidate region.

[0102] Through this step, feature value data including a second feature value indicating a time change of the first feature value or position information is calculated, and the information is output to a module for executing step S106 (eg, analysis module 2804).

[0103] As a suitable example, an example has been described in which the second eigenvalue is calculated based on data for the past 12 hours, but the present disclosure is not limited thereto, and the second eigenvalue may be calculated based on data for the past 4 hours, 6 hours, etc. In addition, an example has been described in which the amount of change in diameter and the average amount of movement are calculated, but the present disclosure is not limited thereto, and the amount of change in the intensity eigenvalue may be calculated, or the variance of the data may be calculated instead of the average value of the amount of change or difference based on linear approximation.

[0104] (Step S106: Analysis Step)

[0105] Step S106 is an analysis step, and in this step, information about the state of the cells in the cell candidate region is acquired by using the feature values ​​calculated in steps S105 and S104. In this case, the information about the state of the cells in the cell candidate region is, for example, a binary value indicating whether the cell candidate region is in a state of a living cell, a dead cell, or a non-cell.

[0106] In this step, a pre-trained trained model is used to use the eigenvalue vector formed by the first eigenvalue obtained in step S104 and the second eigenvalue obtained in step S105 as input, thereby outputting information on the state of cells in each cell candidate region.

[0107] Here, the trained model is a machine learning model trained by using the result of labeling processing based on information about fluorescence intensity as teaching data, the information about fluorescence intensity including at least information about temporal changes in fluorescence intensity.

[0108] The specific method for generating a trained model is described below.

[0109] (Process used to generate trained models)

[0110] Figure 6 FIG1 is a flowchart illustrating the process of generating a trained model for the analysis step in the first embodiment. The following steps are performed on cells for training.

[0111] (Step S601: Acquisition of Time-Series Cell Image Group)

[0112] Step S601 is an image acquisition step of acquiring a time series cell image group for training a trained model.

[0113] In step S101 above, phase contrast images captured at multiple consecutive different timings are used as cell images to obtain a time-series cell image set. However, in this step, fluorescence images of the cell samples used for training are also acquired at the corresponding timings. The phase contrast images and fluorescence images collected at these timings can be referred to as a "time-series phase contrast image set" and a "time-series fluorescence image set," respectively.

[0114] In this case, when generating a trained model that outputs a binary value indicating whether a cell candidate region is in a live cell, dead cell, or non-cell state, a phase contrast image is acquired by phase contrast observation of a cell sample to which a reagent that exhibits fluorescence characteristics in both live cells and dead cells has been added, and a fluorescence image is acquired by fluorescence observation of the cell sample. For example, a reagent that exhibits fluorescence characteristics in live cells and a reagent that exhibits fluorescence characteristics in dead cells, each having different fluorescence characteristics, can be used to distinguish live cells from dead cells.

[0115] In another case, when both live cells and dead cells are stained with a fluorescent dye of the same color, an image can be acquired by photographing a cell sample to which a staining reagent for dead cells and a staining reagent for live cells are added at different timings.

[0116] exist Figure 7 In the figure, image set 702 is shown as an example. Using the same fluorescent dye for both live and dead cells, cells were first plated with a reagent for fluorescently staining dead cells, and images were taken up to 48 hours later. Then, a reagent for fluorescently staining live cells was added to the cells, and images were taken up to 51 hours later, resulting in corresponding phase contrast and fluorescence images of the cells. The image taken immediately after plated is set as the image for incubation time 0 hours, and phase contrast images I700 to I751 and fluorescence images FL700 to FL751 were acquired at one-hour intervals until 51 hours later.

[0117] Through this step, a time series phase contrast image group and a time series fluorescence image group of the cell sample corresponding to the corresponding time are acquired, and the information is output to the module for executing step S602.

[0118] (Step S602 to Step S605)

[0119] The processing steps of step S602 to step S605 are the same as the processing steps of step S102 to step S105 mentioned above applied to the time series phase contrast image group acquired in step S601, and therefore description thereof is omitted.

[0120] Through the processing steps from step S602 to step S605, the time series phase contrast image group obtained in step S601 is calculated as follows: Figure 5 , and outputs the information to a module for executing step S606.

[0121] (Step S606)

[0122] Step S606 is a fluorescence intensity feature value calculation step for calculating feature values ​​related to fluorescence intensity in the time series cell candidate region group based on the fluorescence image acquired in step S601 and the corresponding cell candidate regions and region tracking numbers acquired in steps S602 to S605.

[0123] First, the average fluorescence intensity in each candidate cell region is calculated from the corresponding fluorescence image acquired in step S601. In this case, when the fluorescence image is an RGB color image, the average intensity value of the color component corresponding to the fluorescent dye used to stain the cell sample can be calculated. As described in step S601, when the sample is stained with a green fluorescent dye, the average intensity value of the G component is calculated.

[0124] exist Figure 8 In FIG. 8 , a table 801 is shown as an example, in which the average value of the fluorescence intensity in each cell candidate region is calculated. The average value data of the fluorescence intensity is added to the table 801 as shown in FIG. Figure 5 Such a characteristic value data set calculated from the phase contrast image is shown in . In the following, the average value of the fluorescence intensity is simply referred to as "fluorescence intensity".

[0125] In this case, when Figure 7 When a cell sample in which dead cells and live cells are stained with the same color fluorescent dye at different timings is used as in the example of , each fluorescence intensity calculated from the fluorescent images FL749 to FL751 is obtained by adding the fluorescence intensity due to the staining of dead cells to the fluorescence intensity due to the staining of live cells. Therefore, it is preferable to obtain the fluorescence intensity of each live cell by subtracting the fluorescence intensity due to the staining of dead cells from the fluorescence intensity calculated from the fluorescent image after the staining of live cells. For example, in Figure 7 In the example of , the fluorescence intensity of each cell candidate region calculated from the fluorescence image FL748 at the time point of 48 hours can be subtracted from the fluorescence intensity of the same cell candidate region calculated from the fluorescence images FL749 to FL751 at 49 hours or later. Figure 9 , a graph 901 showing the transition of fluorescence intensity over time in each time series cell candidate region group in a state subjected to the above-mentioned subtraction process is shown. The horizontal axis indicates the number of culture days, the vertical axis indicates the fluorescence intensity, and each line represents each time series cell candidate region group.

[0126] Then, by using Figure 8The fluorescence intensity characteristic value of each time series cell candidate region group is calculated based on the fluorescence intensity of each cell candidate region calculated in Table 801. Here, the fluorescence intensity characteristic value is a characteristic value used to assign a label related to the state of the cells in the cell candidate region (such as "live cell", "non-cell", "dead cell" or "cell with a change in life or death state") to each time series cell candidate region group in the labeling process described later. Therefore, it is expected that the fluorescence intensity characteristic value is a characteristic value that is useful for distinguishing these from each other. For example, four types of scalar values ​​are calculated in the cell candidate region corresponding to each time series cell candidate region group, namely, "fluorescence intensity at the beginning of dead cell staining", "fluorescence intensity at the end of dead cell staining", "maximum value of the change in fluorescence intensity due to dead cell staining" and "maximum value of fluorescence intensity due to live cell staining". For each of the "fluorescence intensity at the beginning of dead cell staining" and the "fluorescence intensity at the end of dead cell staining", data on the fluorescence intensity of the cell candidate region in the cell image at the corresponding timing can be extracted according to the timing at which the live cells are stained. For example, in Figure 7 In the example, the fluorescence intensities at the corresponding timings of 0 hours and 48 hours are extracted. Regarding the "maximum value of the change in fluorescence intensity due to dead cell staining," the change in fluorescence intensity can be obtained by linearly approximating the fluorescence intensity data set for the past 12 hours before any timing, using the same method as described in step S105, and setting its slope to the change. The change in fluorescence intensity of the time series cell candidate region group at the corresponding timing is calculated, and its maximum value is obtained. As the "maximum value of fluorescence intensity due to live cell staining," the maximum value can be obtained from the fluorescence intensity data after the timing at which live cells are stained.

[0127] exist Figure 10 In the example, table 1000 is shown, wherein according to Figure 8 The fluorescence intensity feature value of the corresponding time series cell candidate region group is calculated based on the data of the corresponding cell candidate region including the fluorescence intensity shown in . The fluorescence intensity feature value is calculated for each time series cell candidate region group having the same tracking number.

[0128] As described above, through this step, the feature value related to the fluorescence intensity is calculated for the cell candidate region corresponding to each time series cell candidate region group, and the information is output to the module for executing step S607.

[0129] While the example in which four types of scalar values ​​are calculated as fluorescence intensity feature values ​​has been described, the scalar values ​​calculated as fluorescence intensity feature values ​​are not limited thereto. For example, statistics such as the mean, deviation, minimum, and maximum values ​​of the fluorescence intensity due to dead cell staining in each time-series cell candidate region group may be calculated.

[0130] (Step S607)

[0131] Step S607 is a labeling process step in which information about the state of the cell is assigned to each time series cell candidate region group based on the fluorescence intensity feature value calculated in step S606. Here, the information about the state of the cell is used as a basic fact label for training the trained model described later.

[0132] First, the unsupervised clustering method is used to Figure 10 The fluorescence intensity feature value data set obtained in step S606 shown in FIG is divided into data sets of a freely set number of clusters. Here, it is expected that each time series cell candidate region group will be any of the four types: "live cells," "non-cells," "dead cells," and "cells with altered life-death states." Therefore, when the freely set number of clusters is set to four, well-known unsupervised clustering methods can be applied. For example, hierarchical clustering can be applied. The present disclosure is not limited to this, and methods such as K-means can also be applied.

[0133] Subsequently, a labeling process is performed to determine whether each cluster obtained through the division performed by unsupervised clustering corresponds to a "live cell," "non-cell," "dead cell," or "cell with a life-or-death state change." This labeling process can be performed according to a rule based on the average value of each piece of fluorescence intensity feature value data for each cluster. First, when the time-series cell candidate region group is "dead cells," it is expected that both the "fluorescence intensity at the start of dead cell staining" and the "fluorescence intensity at the end of dead cell staining" have large values. Therefore, in the data set of fluorescence intensity feature values ​​belonging to the cluster, the average value of the "fluorescence intensity at the start of dead cell staining" and the "fluorescence intensity at the end of dead cell staining" is calculated, and the cluster with the largest average value is identified as a "dead cell." Subsequently, when the time-series cell candidate region group is "cell with a life-or-death state change," it is expected that there will be temporal changes in fluorescence intensity due to dead cell staining. Therefore, in the data set of fluorescence intensity feature values ​​belonging to the cluster, the average value of the "maximum value of the change in fluorescence intensity due to dead cell staining" is calculated, and the cluster with the largest average value is identified as a "cell with a life-or-death state change." Furthermore, when the time-series cell candidate region group is "live cells," the "fluorescence intensity due to live cell staining" is expected to be large. Therefore, the average value of the "maximum fluorescence intensity due to live cell staining" is calculated for the data set of fluorescence intensity feature values ​​belonging to each cluster, and the cluster with the largest average value is identified as a "live cell." Finally, the remaining clusters are identified as "non-cells."

[0134] The method of labeling each cluster based on a rule is not limited to the method mentioned above. For example, instead of labeling in the order of "dead cells," "cells with a change in life-death state," "live cells," and "non-cells," labeling can be performed in the order of "cells with a change in life-death state," "dead cells," "live cells," and "non-cells." It is also expected that "non-cells" will be low in both "live cell staining intensity" and "dead cell staining intensity." Therefore, a process for assigning the label "non-cell" can be included so that, in the data set of fluorescence intensity feature values ​​belonging to the cluster, the average value of the "fluorescence intensity at the end of dead cell staining" and the "maximum value of fluorescence intensity due to live cell staining" is calculated to determine that the cluster with the smallest average value is "non-cell."

[0135] Through the above-mentioned processing, it is determined whether each cluster corresponds to a "live cell", "non-cell", "dead cell" or "cell with a change in life-death state", and each time series cell candidate region group belonging to each cluster is assigned a label corresponding thereto as information on the state of the cell. In addition, referring to Figure 11 Regional tracking numbers and labels can also be sent to Figure 8 A label is assigned to each cell candidate region in such a feature value data set of the corresponding cell candidate region shown in .

[0136] Figure 11 is used to illustrate the application of the above-mentioned marking process to Figure 10 The table shows the results of the fluorescence intensity feature value data set of the time series cell candidate region group. The label column in Table 1100 indicates whether each group is "living cell", "non-cell", "dead cell" or "cell with life-death state change", and the labels have been converted into scalar values ​​"0", "1", "2" and "3" respectively. In addition, refer to Figure 11 The area code and label, Figure 12 Table 1200 is used to show that labels are also assigned to Figure 8 A table of results of corresponding cell candidate regions. Figure 13 is a collection of graphs drawn so that the results of the marking process are divided into Figure 9 Graphs are used to illustrate the temporal transition of the fluorescence intensity of each time-series cell candidate region group. Graphs 1301, 1302, 1303, and 1304 respectively illustrate the temporal transition of the fluorescence intensity of the corresponding time-series cell candidate region groups labeled "live cells," "dead cells," "cells undergoing a life-death state transition," and "non-cells."

[0137] As described above, through this step, the feature value data group of each cell candidate region to which the information on the state of the cell has been assigned is output to the module for executing step S608.

[0138] (Step S608: Generation of a trained model)

[0139] Step S608 is a trained model generating step of generating a trained model based on the feature value data group of each cell candidate region to which the information on the state of the cell acquired in step S607 has been assigned.

[0140] In this step, a trained model is generated by training a machine learning model using a feature vector formed by corresponding feature values ​​calculated from the phase contrast image as input and using the label assigned in step S607 as a teacher to output a binary value indicating whether the cell candidate region is in a state of a living cell, a dead cell, or a non-cell. Figure 12 In the example of FIG, the corresponding feature values ​​calculated from the phase contrast image are diameter, average intensity, minimum intensity, intensity deviation, average shift, and diameter change. Furthermore, the trained model is a model structure and parameters optimized based on training data such that a binary value indicating a ground truth data label is obtained when a feature value vector is used as input.

[0141] In this case, about Figure 12 In the data obtained as shown in Figure 1, when observing temporal changes in the same cell, data assigned the label "2" indicates "cells experiencing a change in life-or-death status" and is likely to contain ambiguous information regarding whether the cell is alive or dead. This ambiguous data degrades the performance of the machine learning model used to classify live and dead cells. Therefore, data excluding the data assigned the label "2" is used as training data to generate a trained model.

[0142] It is sufficient for a machine learning model to use a eigenvalue vector as input to perform classification into a freely set number of classes, and as an example thereof, a method such as XGBoost is used. The present disclosure is not limited to this, and models such as neural networks, support vector machines (SVM), random forests, LightGBM or Gaussian mixture models can also be used. In addition, methods for optimizing the structures and parameters of these models based on these training data can also be performed by well-known optimization methods. For example, in the case of XGBoost, optimization is performed by methods such as grid search or Bayesian optimization.

[0143] When performing binary classification into "live cells" and "other" states, rather than "live cells," "dead cells," and "non-cell" states, anomaly detection-based models can also be used. For example, a model that fits a probability density distribution, such as kernel density estimation, can be used. Anomaly detection-based models are suitable for binary classification when there is a very large amount of training data for the other class.

[0144] The method of generating a trained model in the analysis step in the first embodiment has been described above.

[0145] It has been described that the process of obtaining the Figure 7 Although an example of such an image is obtained in a freely selected culture container 701 shown in FIG, when a sample or image of a different field of view on which staining and cell imaging have been performed can be obtained in a similar process, it is also suitable to use these images to generate a trained model. In that case, the processing steps from step S601 to step S607 are repeatedly performed on data of different samples or different fields of view, and the data are converted into the image as shown in FIG. Figure 12 It is sufficient to compile such data obtained after each iteration shown in into one set of training data to then generate a trained model.

[0146] (Step S106: Analysis Step)

[0147] In step S106, Figure 5 This feature value vector of each cell candidate region shown in is input to the trained model generated in steps S601 to S608, and information about the state of the cells in each cell candidate region is obtained. Specifically, in the case of the trained model obtained in the example shown in steps S601 to S608, a binary value indicating whether the cell candidate region is in a "live cell", "dead cell" or "non-cell" state can be obtained as an output result. Figure 14 Is used to show that the input Figure 5 A table of binary values ​​obtained by combining the feature value vectors of each cell candidate region. "Live cell," "dead cell," and "non-cell" correspond to the labels "0," "1," and "3," respectively. For example, the cell candidate region C121 has a binary value of "1" corresponding to the label "0," and a determination result is obtained that the cell candidate region C121 is in the "live cell" state. Instead of a binary value, a likelihood value calculated during the process of outputting a binary value from the machine learning model can be obtained. Figure 15 is a table for illustrating an example of obtaining likelihood values ​​instead of binary values.

[0148] As described above, through step S106 , the eigenvalue vector formed by the first eigenvalue obtained in step S104 and the second eigenvalue obtained in step S105 is input to the trained model to obtain information on the state of cells in each cell candidate region.

[0149] (Output of results)

[0150] The cell image analysis method may further include a display control step of displaying table information (such as a table) including output results from the trained model for corresponding cell candidate regions on a display. Figure 14 Table 1400 or Figure 15 1500), and a graph showing the time transition of the binary value or likelihood value of the corresponding time series cell candidate region group. Figure 14 or Figure 15 In addition to the table information shown in, the output result from the trained model is also preferably displayed in the format of a time-lapse image of a cell candidate area, a time-lapse video generated based on the time-lapse image, or a heat map image. For example, a time-lapse image of the outline of the cell in the cell candidate area is drawn on each timing phase contrast image by changing the display mode according to the determination result from the trained model. As used herein, the wording "changing the display mode" refers to a requirement that the display mode is different for each determination result in terms of the degree to which the determination result from the trained model can be grasped. The example of changing the display mode includes color coding and decoding the outline, changing the thickness of the outline, changing the type of the outline (the example of the type of line includes a solid line, a broken line, and a single dot dash line) or a combination of these. Instead of showing the outline of the cell, it is conceivable to indicate the cell area with an arrow, color the cell area, or illustrate the major and minor diameters of the cell area. It is also conceivable to allow the determination result to be grasped by changing the intensity, coloring, or arrows of the outline. In addition, as a heat map image, a heat map image in which the width direction indicates time, the height direction indicates the tracking number of the region, and the numerical value of each pixel represents a binary value or a likelihood value can be displayed. In the case of such a trained model that determines the state of "live cells", "dead cells" or "non-cells" as described in the example, it is also suitable to calculate the live cell rate ((number of live cells) / ((number of live cells)+(number of dead cells))) etc. at each timing and display the time transition of the live cell rate.

[0151] As described above, according to this embodiment, it is possible to present a determination result on the state of a cell to the user by using a trained model based on more reliable labeled data.

[0152] (First Modification Example of First Embodiment: Position Alignment)

[0153] In the first embodiment, the cell images I00 to I51 have been described as examples of images taken with the same field of view. Instead of using images taken with a field of view including exactly the same cell region, when using images taken with a field of view including some common cell regions or when it is expected that the shooting position will move due to vibration of the device, etc., it is desirable to perform position alignment between multiple images in the time series cell image group. Position alignment can be performed in steps such as step S101, step S102 or step S103, or can be performed in a separate step. Position alignment is only required to be performed between images between successive timings, and well-known position alignment techniques such as template matching can be used. Not only template matching can be used, but also a position alignment method (such as SIFT or SURF) in which feature points are extracted and an affine transformation matrix is ​​estimated based on feature point matching can be used. In addition, instead of using phase contrast images to perform position alignment, it is also possible to use a method such as Figure 3 Position alignment is performed using such mask images as shown in .

[0154] According to this modification example, even when a misalignment of the visual field occurs between cell images at different timings in a time-series cell image group, a trained model can be generated and information related to the state of the cell can be acquired by using the trained model.

[0155] (Second Modification Example of First Embodiment: Case of Sample in Which Live Cells and Dead Cells Are Stained with Different Dyes)

[0156] In the first embodiment, in the example of the trained model generation method from step S601 to step S608, an example has been described in which a fluorescent image of a cell sample is acquired in which live cells and dead cells are stained with a fluorescent dye of the same color at different timings. The present disclosure is not limited thereto, and a fluorescent image of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors (i.e., fluorescent dyes having at least one different excitation wavelength and fluorescence wavelength) may be used for trained model generation.

[0157] In that case, in Figure 7 In the example, a fluorescence image corresponding to each of live and dead cells can be acquired at each timing. Fluorescence images are two types of grayscale images corresponding to live and dead cells, respectively. When the fluorescence image is an RGB color image, the color component corresponding to the fluorescent dye used to stain the cell sample can be extracted. For example, when the cell sample is stained with a red dye for live cells and a green dye for dead cells, images of the R and G components are extracted from the RGB color image. Figure 16This diagram illustrates an example (image 1602) of acquiring phase contrast images, fluorescence images corresponding to the dead cell staining, and fluorescence images corresponding to the live cell staining in a cell sample in which live cells and dead cells are stained with different colored fluorescent dyes. Fluorescent images FL1600a to FL1651a corresponding to the dead cell staining and fluorescent images FL1600b to FL1651b corresponding to the live cell staining have been acquired, and both correspond to phase contrast images I1600 to I1651 at corresponding timings.

[0158] When Figure 16 When fluorescent images corresponding to live cell staining and dead cell staining in the culture container 1601 are obtained as shown in the image 1602, in step S606, the average of the intensity values ​​in each cell candidate region of each fluorescent image is calculated, and based on the corresponding fluorescence intensities, as shown in FIG. Figure 11 The fluorescence intensity feature value of each time series cell candidate region group is calculated as shown in . The first embodiment has been described above by taking the calculation of "fluorescence intensity at the start of dead cell staining", "fluorescence intensity at the end of dead cell staining", and "maximum value of the change in fluorescence intensity due to dead cell staining" as an example of the feature value expressing the time change of fluorescence intensity due to dead cell staining. Figure 11 . Also, as in this modified example, when a fluorescence image starting from 0 hours into the culture period is acquired with respect to live cell staining, it is preferable to calculate a characteristic value expressing the temporal change in fluorescence intensity due to live cell staining. Specifically, in the same manner as in the case of dead cell staining, the "fluorescence intensity at the start of live cell staining," the "fluorescence intensity at the end of live cell staining," and the "minimum value of the change in fluorescence intensity due to live cell staining" can be calculated.

[0159] Thereafter, in step S607 and subsequent processing steps, in the same manner as in the first embodiment, labeling processing is performed by using the calculated fluorescence intensity feature values ​​to generate a trained model.

[0160] As described above, according to this modification example, even when a fluorescent image of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors is acquired, a trained model can be generated and information on the state of the cells can be acquired by using the trained model.

[0161] (Third Modification Example of First Embodiment: Case of Sample in Which Living Cells and Dead Cells Are Stained with Different Dyes)

[0162] In the first embodiment, in the example of the trained model generation method from step S601 to step S608, an example has been described in which a fluorescent image of a cell sample is obtained in which live cells and dead cells are stained with fluorescent dyes of the same color at different timings. The present disclosure is not limited to this, and in step S601, a fluorescent image of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors may be used. Examples of reagents for staining cells include nuclear staining reagents and cell membrane staining reagents.

[0163] In that case, in Figure 7 In the example, a fluorescence image corresponding to each of live and dead cells can be acquired at each timing. Fluorescence images are two types of grayscale images corresponding to live and dead cells, respectively. When the fluorescence image is an RGB color image, the color component corresponding to the fluorescent dye used to stain the cell sample can be extracted. For example, when the cell sample is stained with a red dye for live cells and a green dye for dead cells, images of the R and G components are extracted from the RGB color image. Figure 27 This figure illustrates an example of acquiring phase contrast images, fluorescence images corresponding to the dead cell staining, and fluorescence images corresponding to the live cell staining in a cell sample in which live cells and dead cells are stained with different colors of fluorescent dyes. Fluorescent images FL2700a to FL2751a corresponding to the dead cell staining and fluorescent images FL2700b to FL2751b corresponding to the live cell staining have been acquired, and both correspond to phase contrast images I2700 to I2751 at corresponding timings.

[0164] When Figure 27 When fluorescent images corresponding to live cell staining and dead cell staining in the culture container 2701 are obtained as shown in the image 2702, in step S606, the average of the intensity values ​​in each cell candidate region of each fluorescent image is calculated, and based on the corresponding fluorescence intensities, as shown in FIG. Figure 11 Calculate the fluorescence intensity feature value for each time series cell candidate region group as shown in . In addition, calculate the "fluorescence intensity at the start of dead cell staining", "fluorescence intensity at the end of dead cell staining", and "maximum value of the change in fluorescence intensity due to dead cell staining" as feature values ​​expressing the temporal change in fluorescence intensity due to dead cell staining, as shown in Figure 11 As shown in . Regarding cell staining, temporal variation is small, so the fluorescence intensity at a specific time point or the average value of the time series data of fluorescence intensity can be taken.

[0165] Thereafter, in step S607 and subsequent processing steps, in the same manner as in the first embodiment, labeling processing is performed by using the calculated fluorescence intensity feature values ​​to generate a trained model.

[0166] As described above, according to this modification example, even when a fluorescent image of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors is acquired, a trained model can be generated and information on the state of the cells can be acquired by using the trained model.

[0167] It is also suitable to use fluorescent images of cell samples in which living cells and dead cells are stained with fluorescent dyes of different colors, and the same method can be used to generate a trained model and obtain information about the state of the cells by using the trained model.

[0168] (Fourth Modification Example of the First Embodiment: Unsupervised Clustering in Steps)

[0169] In the first embodiment, a method of labeling processing has been described in which the fluorescence intensity feature value data group of the corresponding time series cell candidate region group is divided into four types of clusters by unsupervised clustering, and each cluster is labeled based on a rule. The present disclosure is not limited to this, and the labeling processing can be performed step by step in step S607. The step-by-step labeling processing is effective when the fluorescence intensity feature values ​​that are particularly effective for classification using unsupervised clustering and the classification difficulty are easy to predict. Specifically, in the example of assigning four types of labels, namely "living cells", "non-cells", "dead cells", and "cells with a change in life and death status", based on, for example Figure 9 From the temporal transition of fluorescence intensity shown in , it can be expected that classification between "live cells" and "non-cells" is difficult, while classification between "dead cells" or "cells with altered life and death states" and others is relatively easy.

[0170] Figure 17 This diagram illustrates the data flow obtained when labeling processing is applied stepwise to a set of fluorescence intensity feature value data. In the same manner as in the first embodiment, this diagram illustrates an example in which four types of labels—"live cells," "non-cells," "dead cells," and "cells with a life-death state change"—are assigned to a set of time-series cell candidate regions.

[0171] First, the fluorescence intensity characteristic value data group D1700 is obtained before the labeling process is performed, and is compared with the fluorescence intensity characteristic value data group D1700. Figure 10 Corresponding to data groups such as tables.

[0172] Step S1701 is the first labeling step for classifying the fluorescence intensity feature value data set D1700 into three types of data sets: "live cells or non-cells," "dead cells," and "cells with altered life-death states." In the first labeling step, an unsupervised clustering method is applied with the number of clusters set to three, and each cluster is then labeled based on a rule. In this case, in the unsupervised clustering in the first labeling step, it can be easily expected that Figure 10 Among the fluorescence intensity feature values ​​shown in , the three types of feature values, "fluorescence intensity at the beginning of dead cell staining", "fluorescence intensity at the end of dead cell staining" and "maximum value of the amount of change in fluorescence intensity due to dead cell staining", contribute greatly to classification. Therefore, it is also appropriate to perform unsupervised clustering by using only these three types of feature values. Rule-based labeling can be performed in the same manner as the method of assigning labels "dead cells" and "cells with a change in life or death status" described in the first embodiment, and the remaining one cluster is labeled as "live cells or non-cells". Through the first labeling processing step mentioned above, the fluorescence intensity feature value data group D1700 is divided into three types of data groups D1701a, D1701b and D1701c, namely, "live cells or non-cells", "dead cells" and "cells with a change in life or death status".

[0173] Step S1702 is a second labeling process step for dividing the data set D1701a labeled "live cells or non-cells" into two types of data sets: "live cells" and "non-cells". In the second labeling process step, an unsupervised clustering method is applied with the number of clusters set to two, and then each cluster is labeled based on a rule. In this case, in the unsupervised clustering in the second labeling process step, it can be easily expected that Figure 10 Among the fluorescence intensity feature values ​​shown in , the feature value of "maximum fluorescence intensity due to staining for live cells" contributes significantly to classification. Therefore, unsupervised clustering using only this one type of feature value is also suitable. Rule-based labeling can be performed in the same manner as the method of assigning the label "live cells" described in the first embodiment, and the remaining cluster can be labeled "non-cells." Through the second labeling process mentioned above, the fluorescence intensity feature value data set D1701a is divided into two types of data sets D1702a and D1702b, namely, "live cells" and "non-cells."

[0174] Finally, the data groups D1701b, D1701c, D1702a and D1702b with determined labels are combined into a labeled fluorescence intensity feature value data group D1701, and a data group to which basic fact labels are assigned can be obtained, as in the first embodiment. Figure 11 、 Figure 12 Like in.

[0175] The processing steps of step S608 and subsequent processing steps are the same as those of the first embodiment.

[0176] As described above, according to this modification example, when fluorescence intensity feature values ​​that are particularly effective for classification using unsupervised clustering and the difficulty of classification are easy to predict, a trained model can be generated using training data with ground truth labels having a higher degree of reliability.

[0177] (Fifth Modified Example of the First Embodiment: Data Cleaning of "Cells with Changed Life and Death Status")

[0178] In step S608 in the first embodiment, an example has been described in which, in the process of generating a trained model for classifying “living cells,” “dead cells,” and “non-cells,” training is performed by excluding data to which the label “cells with a change in life or death status” has been assigned from the training data.

[0179] The present invention is not limited to this, and when a training model is generated, the labels of some cell candidate regions in the data that have been assigned the label "cells with a life-or-death state transition" can be changed to the labels "live cells" or "dead cells" and used as training data. Specifically, the change in fluorescence intensity in each time series of cell candidate region groups assigned the label "cells with a life-or-death state transition" is used to determine each of the culture time slots in which cells are in the live cell state, the culture time slots in which the state transitions from live to dead cells, and the culture time slots in which cells are in the dead cell state. Figure 18 It is an example showing the time transition of the fluorescence intensity and the time transition of the amount of change in fluorescence intensity of a time series cell candidate region group to which the label "cells with a change in life or death state" has been assigned. Graph 1800 shows the time transition of the fluorescence intensity, and graph 1801 shows the time transition of the amount of change in fluorescence intensity. The time transition of the fluorescence intensity of "cells with a change in life or death state" involves a significant increase in intensity from a low intensity state when the state transitions from a living cell to a dead cell. Therefore, in the time transition of the amount of change in fluorescence brightness, the timing at which the amount of change reaches a maximum value can be set to at least a reference timing at which the state transitions from a living cell to a dead cell, so as to determine the time slot in which the amount of change before the reference timing is equal to or less than the threshold as the time slot in the "living cell" state, and the time slot in which the amount of change after the reference timing is equal to or less than the threshold as the time slot in the "dead cell" state. The threshold is set to, for example, "5". In Figure 18 In the example, time slot 1801a is a time slot for the "live cell" state, and time slot 1801c is a time slot for the "dead cell" state. The label of the data for each cell candidate region corresponding to the time slot is replaced with the label "live cell" or "dead cell." Instead of setting a threshold, the full width at half maximum relative to a reference position can be used to determine the time slot.

[0180] According to this modified example, a trained model can be generated using training data with an increased amount of training data related to “live cells” and “dead cells”.

[0181] (Sixth Modification Example of the First Embodiment: Data Cleaning of “Non-Cells”)

[0182] In the first embodiment, the labeling process for assigning the labels "live cells," "non-cells," "dead cells," and "cells with altered life-or-death status" has been described. In this modified example, a data cleaning method is described that excludes data in a more ambiguous state from the labeling process results.

[0183] As a specific example, the following describes a method for cleaning data corresponding to the label "non-cell" in the first embodiment.

[0184] As described above, in addition to the cell region, "non-cells" are also assumed to be objects such as cell fragments and scratches or adhered substances on culture containers, and it is expected that basic fact labels including "cell fragments" and "scratches or adhered substances" that distinguish between them will be assigned to the labeled data for the machine learning model. For example, "cell fragments" are expected to take more diverse forms than "scratches or adhered substances", so uniform training as "non-cells" will lead to degradation of the performance of the machine learning model. Therefore, this labeling processing for distinguishing "cell fragments" and "scratches or adhered substances" from each other can be further performed on the data labeled as "non-cells". Specifically, as described in the fourth modified example of the first embodiment, unsupervised clustering and rule-based labeling using specific fluorescence brightness feature values ​​are performed. For example, using Figure 10 Unsupervised clustering was performed on the "fluorescence brightness at the end of dead cell staining" among such fluorescence brightness feature values ​​shown in , where the number of clusters was set to two. The average values ​​of the "fluorescence brightness at the end of dead cell staining" in the respective clusters were then compared with each other, and the label "non-cell" was assigned to the data belonging to the cluster with the smaller average value, while the label "cell debris" was assigned to the data belonging to the other cluster. Figure 19 is used to illustrate the application of data cleaning in this modified example to Figure 13 Graph 1304 shows a collection of graphs of the results of such a data set, including a time series of cell candidate region groups assigned the label “non-cell”. Graph 1901 shows the temporal transition of the fluorescence intensity of the time series cell candidate region group assigned the label “non-cell”, and graph 1902 shows the temporal transition of the fluorescence intensity of the time series cell candidate region group assigned the label “cell debris”.

[0185] In the trained model generation step of step S608 , training of the machine learning model is performed by using training data excluding data to which the label “cell debris” has been assigned.

[0186] This modification example can be performed in combination with the marking process in each of the first embodiment and other modification examples as a post-processing thereof. Whether or not the data cleaning in this modification example is performed can be displayed on the display as having undergone such a process. Figure 13 Such a graph of the time transition of the fluorescence intensity of the marker process shown in et al. is then determined by indication by the user.

[0187] (Seventh Modification Example of First Embodiment: Analysis of Information Indicating Whether a Cell Candidate Region is a Differentiation Region)

[0188] It is also suitable to analyze information indicating whether the cell candidate region is a differentiated region as the state of the cell candidate region in step S106. That is, in this modified example, a binary value indicating whether the cell candidate region is a differentiated region is obtained in step S106 by using the feature values ​​calculated in steps S105 and S104.

[0189] It is known that when culturing cells with differentiation potential, such as embryonic stem cells (ES cells), induced pluripotent stem cells (iPS cells), and somatic stem cells, cells in a further differentiated state may be generated when culture conditions are not suitable. A differentiation zone refers to a region that exhibits this further differentiated state.

[0190] When generating a learning model for analyzing information indicating whether a candidate cell region is a differentiated region, a set of time-series fluorescence images obtained by fluorescently staining with undifferentiation markers or differentiation markers is acquired. Fluorescent staining reagents for undifferentiation markers are used for ES cells or iPS cells, such as rBC2LCN or various pluripotent stem cell markers, and for somatic stem cells, fluorescent staining reagents for corresponding somatic stem cell markers are used.

[0191] As differentiation markers, in the case of ES cells or iPS cells, three germ layer markers can be selected, while in the case of somatic stem cells, markers corresponding to further differentiated cells can be selected with reference to a differentiation lineage map.

[0192] The other steps are the same as those described in the first embodiment, and thus description thereof is omitted. According to this modification example, as information on the state of cells, a binary value or the like indicating whether a cell candidate region is a differentiated region may be analyzed.

[0193] (Eighth Modified Example of First Embodiment: Unsupervised Clustering is Performed Using Time Series Data as Input)

[0194] In the first embodiment, the description has been made based on Figure 11The present disclosure is not limited thereto, and such an unsupervised clustering method that uses time series data as input to divide the time series data into a freely set number of clusters may be used. For example, a method such as TimeSeriesKMeans may be used. When a method using time series data as input (such as TimeSeriesKMeans) is used, as shown in FIG. Figure 8 Such a time series data set of fluorescence intensity in the corresponding time series cell candidate region group shown in is used as subject data for unsupervised clustering.

[0195] When it is difficult to predict the feature values ​​that are particularly useful for unsupervised clustering (e.g. Figure 10 This modification example is effective when the fluorescent brightness at a specific timing or the statistical value of the change amount of the fluorescent brightness is shown).

[0196] (See Example: Evaluation of a Trained Model)

[0197] In the first embodiment, an example of training a trained model has been described, and the trained model can be evaluated by preparing a data set for evaluation that is different from the data used for training. In this case, as the data set for evaluation, a data set extracted by cutting out a portion of the data set for training can be used. For example, data is randomly extracted so that the ratio of the number of data pieces used for training to the number of data pieces used for evaluation is 8:2. When the data set subjected to training and evaluation for the trained model is a data set obtained with respect to a plurality of different culture containers, the data set can be divided into a data set for training and a data set for evaluation based on the culture container, rather than randomly extracted from the entire data set.

[0198] This reference example describes an example of evaluating a trained model to be performed when generating a live cell determination model for determining whether a cell candidate region is a live cell. A machine learning model based on anomaly detection can be trained as the live cell determination model, which performs two-class classification into the "live cell" state and the "other" state, as described in the first embodiment.

[0199] Evaluation can be performed by comparing the ground truth labels assigned to the data set for evaluation through the processing based on steps S601 to S607 with the information on the states of cells in the cell candidate region acquired based on steps S101 to S106. More specifically, it is sufficient to calculate indicators such as sensitivity (recall), specificity, and F1-score, which are commonly used evaluation indicators in anomaly detection.

[0200] Figure 30is a table showing an example of the evaluation results of the living cell determination model. Figure 15 Such a feature value vector formed by the diameter, average brightness, minimum brightness, brightness deviation, average movement and diameter change of each cell candidate region in the cell image shown in is used as an example of input data of the trained model. Figure 30 In the example of , the accuracies of three types of living cell determination models No. 1, No. 2, and No. 3 obtained by changing the types and combinations of feature values ​​forming input data are compared with each other. Figure 30 , the circular symbols indicate which eigenvalues ​​are included in each living cell determination model as eigenvalues ​​forming the input eigenvalue vector.

[0201] Precision (live cells) and precision (non-live cells) have the same meanings as specificity and sensitivity (recall), respectively, and are commonly used as evaluation metrics for anomaly detection models. Additionally, the F1 score is the harmonic mean of precision and sensitivity (recall), and is commonly used as an evaluation metric for anomaly detection models in the same manner as sensitivity (recall) and specificity.

[0202] Figure 31 3100 is a graph showing the results of evaluating the accuracy (living cells) of the culture time per hour. Figure 30 The models No.1, No.2 and No.3 in Table 3000 correspond to each other. Figure 32 3200 is a graph showing the results of evaluating the accuracy (non-viable cells) of the culture time per hour. Figure 30 Corresponding to models No.1, No.2 and No.3 in Table 3000.

[0203] The cell image analysis method according to the first embodiment may further include a display control step of displaying the evaluation result in this reference example. In that case, it is preferable as Figure 30 、 Figure 31 or Figure 32 The information about the data set used for training and evaluation of the trained model and such numerical values ​​or a graph showing the evaluation results are shown in . The information about the data set is, for example, information indicating culture conditions (such as the number of culture days and seeding density of the cell sample), information indicating shooting conditions (such as the image pickup field of view and exposure time), and information indicating the number of data pieces (such as the total number of data pieces and the number of data pieces for each basic truth label).

[0204] (Second reference example: Evaluation of trained models)

[0205] In the reference example mentioned above, an example has been described in which the evaluation results of three types of live cell determination models obtained by training a machine learning model based on abnormality detection by changing the types and combinations of feature values ​​forming input data are compared with each other.

[0206] In this reference example, as another suitable example, an example is described in which evaluation results of four types of living cell determination models obtained by training a machine learning model using a decision tree based on the CART algorithm by changing the type and combination of feature values ​​forming input data are compared with each other. As an evaluation indicator for evaluating the trained machine learning model, Figure 30 The accuracy (live cells), accuracy (non-live cells) and F1-score were calculated in the same way as in

[15] .

[0207] Figure 33 TABLE 1 is a table showing an example of the results of evaluating the accuracy of the living cell determination model. The accuracies of four types of living cell determination models No. 1, No. 2, No. 3, and No. 4 obtained by changing the type and combination of the feature values ​​forming the input data are compared with each other. Figure 30 Similarly, the circle symbols indicate which eigenvalues ​​are included in the model for each live cell determination as eigenvalues ​​forming the input eigenvalue vector. In this case, the brightness feature is Figure 15 A general term for the average value of brightness, the minimum value of brightness, and the deviation of brightness among the pieces of data shown in .

[0208] Figure 33 This table shows the results of dividing the data set for evaluation into two data sets (one for culture time of 0 to 11 hours, and the other for culture time of 12 hours or more), and then calculating the evaluation index for each data set for each culture time slot. In this way, the evaluation value for each separate culture time slot can be calculated and displayed.

[0209] Figure 34 3400 is a graph showing the results of evaluating the accuracy (living cells) of the culture time per hour. Figure 33 The models No.1, No.2, No.3 and No.4 in Table 3300 shown in FIG. 3 correspond to each other. Figure 35 This is a graph showing the results of evaluating the accuracy (non-viable cells) per hour of culture time. Figure 35 In the graph 3500 shown in FIG, graphs 3501, 3502, 3503 and 3504 are respectively Figure 33The models No. 1, No. 2, No. 3, and No. 4 in Table 3300 shown in FIG. 3 correspond to each other.

[0210] As described above, based on this reference example, the user can grasp the performance of the trained model.

[0211] Used to describe reference examples Figures 30 to 35 The evaluation results of different models for time slots or at predetermined times are shown. In the analysis step in the first embodiment, the trained models can be switched according to time based on these evaluation results. For example, the trained model with the highest F1-score is selected for each time slot. Figure 33 In the example, model No. 2 is applied to the data group with a culture time of 0 to 11 hours, and model No. 4 is applied to the data group with a culture time of 12 hours or more. Figures 30 to 35 The evaluation results are used to predetermine which model should be used for which time slot configuration.

[0212] [Second embodiment]

[0213] (Overview)

[0214] In the first embodiment, a method has been described for obtaining information about the state of a cell by using an eigenvalue vector formed by brightness eigenvalues ​​and morphological eigenvalues ​​calculated from a time series phase contrast image group obtained by observing in time series, or an eigenvalue vector formed by eigenvalues ​​related to temporal changes in brightness eigenvalues ​​and morphological eigenvalues ​​as input to a trained model.

[0215] In a second embodiment, a method is described for obtaining information about the state of a cell by using time series data of brightness feature values ​​and morphological feature values ​​calculated based on a time series phase contrast image group obtained by observing in time series as input to a trained model, wherein the time series data is time-related information.

[0216] (Processing process)

[0217] Figure 20 is a flowchart of the cell information acquisition method according to the second embodiment.

[0218] Steps S2001 to S2004 are identical to steps S101 to S104 in the first embodiment, and thus their description is omitted. Through steps S2001 to S2004, time series data of brightness feature values ​​and morphological feature values ​​for each time series cell candidate region group is acquired and output to a module (e.g., analysis module 2804) for executing step S2005.

[0219] (Step S2005: Analysis Step)

[0220] Step S2005 is a step of acquiring information on the state of cells in the cell candidate region by using the time series data of the brightness feature values ​​and the morphological feature values ​​calculated in steps S2001 to S2004 .

[0221] In this step, a pre-trained model is used to use the time series data of feature values ​​such as the brightness feature value and the morphological feature value calculated in step S2004 as input, thereby outputting information about the state of the cells in the cell candidate region. The information about the state of the cells in the cell candidate region is, for example, a binary value indicating whether the cell candidate region is in a live cell state, a dead cell state, or a non-cell state, as in the first embodiment.

[0222] In the machine learning model of this embodiment, the time series feature value data obtained by extracting the time series data of the feature value for the past fixed time slot before any timing is used as input data. For example, the time series feature value data for the time slots of the past 12 hours is used as input. As a specific example, Figures 21A to 21C The figure shows the results of extracting time series feature value data used as input for the machine learning model at regular intervals during the 12-hour culture period. Figure 21A Table 2100 shown in FIG. 2 is a feature value data set acquired in step S2004 . Figure 21B The table 2101 shown in Figure 21A The result of extracting the time series feature value data at the timing of the culture time 12 hours from the feature value data group of Table 2100 is shown in FIG. The feature value data for the corresponding time slot in each time series cell candidate region group with the same tracking number is extracted, such as the time series feature value data DT001 or DT002. In this embodiment, Figure 21B Each piece of data such as the time series feature value data DT001 and DT002 shown in FIG is input into the trained model to obtain Figure 21C Such results shown in Table 2102 are RT001 (corresponding to DT001) and RT002 (corresponding to DT002), which are binary values ​​indicating whether the cell candidate region is in a live cell, dead cell, or non-cell state.

[0223] The specific method for generating a trained model is described below.

[0224] As in the first embodiment, the trained model is a machine learning model trained by using the result of labeling processing based on information about fluorescence brightness including at least information about temporal changes in fluorescence brightness as teaching data.

[0225] (Process used to generate trained models)

[0226] Figure 22 It is a flowchart for illustrating the process of generating a trained model through the analysis step in the second embodiment.

[0227] Steps S2201 to S2204 are identical to steps S601 to S604 in the first embodiment, and thus their description is omitted. Through steps S2201 to S2204, time series data of the brightness feature values ​​and morphological feature values ​​for each time series cell candidate region group is acquired, and the acquired time series data is output to the module for executing step S2205.

[0228] Steps S2205 and S2206 are the same as steps S606 and S607 in the first embodiment, and therefore their description is omitted. Through steps S2205 and S2206, data in which the basic truth labels related to the states of cells in the cell candidate regions are assigned to the time series data of the brightness feature values ​​and morphological feature values ​​in each time series cell candidate region group is output to the module for executing step S2207.

[0229] (Step S2207: Generation of a trained model)

[0230] Step S2207 is a trained model generating step of generating a trained model based on the feature value data group of each cell candidate region to which the information on the state of the cell acquired in step S2206 has been assigned.

[0231] In this step, a trained model is generated by training a machine learning model that uses the time series data of the feature values ​​in the cell candidate region of the phase contrast image included in each time series cell candidate region group as input and uses the label assigned in step S2206 as a teacher to output a binary value indicating whether the cell candidate region is in a live cell, dead cell, or non-cellular state. The machine learning model can be any machine learning model that uses the time series data of the feature values ​​as input to perform classification into a freely set number of classes, and as an example thereof, a classification model using a neural network such as RNN can be used. The present disclosure is not limited to this, and a neural network model such as LSTM can also be used. A neural network that performs image classification, such as ResNet, can also be trained by treating the time series data of multiple types of feature values ​​as two-dimensional array data.

[0232] The method of generating a trained model through the analysis step in the second embodiment has been described above.

[0233] In step S2005, Figure 21BSuch time series feature value data shown in is input to the trained model generated in steps S2201 to S2207 to obtain information about the states of cells in the cell candidate region.

[0234] As described above, according to this embodiment, it is possible to present a determination result on the state of a cell to the user by using a trained model based on more reliable labeled data.

[0235] [Third embodiment]

[0236] (Overview)

[0237] In the first embodiment, a method has been described for obtaining information about the state of a cell by using an eigenvalue vector formed by brightness eigenvalues ​​and morphological eigenvalues ​​calculated from a time series phase contrast image group obtained by observing in time series, or an eigenvalue vector formed by eigenvalues ​​related to temporal changes in brightness eigenvalues ​​and morphological eigenvalues ​​as input to a trained model.

[0238] In the third embodiment, a method of acquiring information on the state of cells by using a partial cell image obtained by cutting out a cell region corresponding to a cell candidate region from a phase contrast image as an input to a trained model is described.

[0239] (Processing process)

[0240] Figure 23 FIG. 4 is a flowchart of a cell information acquisition method according to the third embodiment.

[0241] Steps S2301 to S2303 are the same as steps S101 to S103 in the first embodiment, so their description is omitted. Through steps S2301 to S2303, for the cell candidate regions, region tracking numbers indicating which cell candidate regions are temporally continuous and are the same cell candidate region are assigned to the mask image and the contour coordinate information corresponding to each cell region, and the information is output to the module for executing step S2304 (e.g., analysis module 2804).

[0242] (Step S2304: Analysis Step)

[0243] Step S2304 is an analysis step of acquiring information on the status of cells in the cell candidate region based on the region tracking number and the cell candidate region acquired in step S2303 .

[0244] In this step, a pre-trained trained model is used to use a partial cell image corresponding to each cell candidate region as input based on the region tracking number and cell candidate region obtained in step S2303, thereby outputting information about the state of the cells in the cell candidate region. Here, the partial cell image is an image obtained by cutting out a rectangular region of a predetermined size centered on the centroid position of the cell candidate region from the cell image. The predetermined size can be any size that includes the target cell candidate region, and for example, a rectangular region of 64x64 pixels is cut out. In addition, as in the first embodiment, the information about the state of the cells in the cell candidate region is, for example, a binary value indicating whether the cell candidate region is in a live cell, dead cell, or non-cell state.

[0245] In the machine learning model of this embodiment, a partial cell image group obtained by extracting partial cell images for a fixed time slot in the past before any timing is used as input data. For example, a partial cell image group for a time slot in the past 12 hours is used as input. As a specific example, Figure 24A and Figure 24B Figure 3 shows the results of extracting a subset of cell images used as input to the machine learning model over a 12-hour culture period. Figure 24A The image 2401 is a partial cell image group cut out from the corresponding cell image based on the region tracking number and cell candidate region obtained in step S2303. For example, at a time of 12 hours of culture time, a partial cell image group (such as the partial cell image groups 2401a and 2401b) is used as an input to the machine learning model. The partial cell image group (such as the partial cell image group 2401a and 2401b) is input to the trained model to obtain the following for each partial cell image group: Figure 24B Such results are shown in Table 2402a (corresponding to 2401a) and Table 2402b (corresponding to 2401b) in Table 2402 of the present invention as binary values ​​indicating whether the cell candidate region is in a living cell, dead cell or non-cell state.

[0246] The specific method for generating a trained model is described below.

[0247] As in the first embodiment, the trained model is a machine learning model trained by using the result of labeling processing based on information about fluorescence brightness including at least information about temporal changes in fluorescence brightness as teaching data.

[0248] (Process used to generate trained models)

[0249] Figure 25 It is a flowchart for illustrating the process of generating a trained model through the analysis step in the third embodiment.

[0250] Steps S2501 to S2503 are the same as steps S601 to S603 in the first embodiment, and thus their description is omitted. Through steps S2501 to S2503, for the cell candidate regions, region tracking numbers are assigned to the mask image and the contour coordinate information corresponding to each cell region, indicating which cell candidate regions are temporally continuous and are the same cell candidate region, and the information is output to the module for executing step S2504.

[0251] Steps S2504 and S2505 are the same processing as steps S606 and S607 in the first embodiment, so their description is omitted. Through steps S2404 and S2405, data in which a basic fact label related to the state of the cell in the cell candidate region is assigned to each cell candidate region is output to the module for executing step S2506 (e.g., the analysis module 2804).

[0252] (Step S2506: Generation of a trained model)

[0253] Step S2506 is a trained model generating step of generating a trained model based on each cell candidate region to which the information on the state of the cell acquired in step S2505 has been assigned.

[0254] In this step, a trained model is generated by training a machine learning model using a set of partial cell images obtained by extracting partial cell images for a fixed time slot in the past before any timing as input, and using the label assigned in step S2506 as a teacher to output a binary value indicating whether the cell candidate region is in a live cell, dead cell, or non-cellular state. As a machine learning model, a neural network such as ConvLSTM that uses time series images as input can be used. A neural network model that classifies three-dimensional volume data, such as 3D-ResNet, can also be trained by treating the set of partial cell images as time series image data as three-dimensional volume data.

[0255] The method of generating a trained model through the analysis step in the third embodiment has been described above.

[0256] In step S2304, Figure 24A Such a partial cell image group shown in is input to the trained model generated in steps S2501 to S2506 to obtain information about the states of cells in the cell candidate region.

[0257] As described above, according to this embodiment, it is possible to present a determination result on the state of a cell to the user by using a trained model based on more reliable labeled data.

[0258] <Cell Information Acquisition Program and Cell Information Acquisition System>

[0259] The present disclosure provides a program for causing a computer to execute the cell image analysis method according to the present disclosure described so far. In addition, the present disclosure provides a medium on which the above-mentioned program is stored in a computer-readable format.

[0260] Figure 26 26 is a block diagram for illustrating a hardware configuration example of an information processing device 2600 capable of executing a program according to the present disclosure.

[0261] The information processing device 2600 has the function of a computer, and for example, the information processing device 2600 may be configured integrally with, for example, a desktop personal computer (PC), a laptop PC, a tablet PC, or a smartphone.

[0262] In order to realize the function as a computer that performs arithmetic operations and stores data, the information processing device 2600 includes a central processing unit (CPU) 2601, a random access memory (RAM) 2602, a read-only memory (ROM) 2603, and a hard disk drive (HDD) 2604. The information processing device 2600 also includes a communication interface (I / F) 2605, a display device 2606, and an input device 2607. The CPU 2601, the RAM 2602, the ROM 2603, the HDD 2604, the communication I / F 2605, the display device 2606, and the input device 2607 are connected to each other via a bus 2610. The display device 2606 and the input device 2607 can be connected to the bus 2610 via a driver device (not shown) for driving these devices.

[0263] exist Figure 26 , the components forming the information processing device 2600 are shown as integrated devices, but some of the functions of these components may be implemented by external devices. For example, the display device 2606 and the input device 2607 may be external devices different from the components that implement the functions of the computer including the CPU 2601.

[0264] The CPU 2601 performs predetermined operations according to programs stored in, for example, the RAM 2602 and the HDD 2604, and also has a function of controlling each component of the information processing device 2600. The RAM 2602 is constructed from a volatile storage medium and provides a temporary storage area required for the operation of the CPU 2601. The ROM 2603 is constructed from a nonvolatile storage medium and stores necessary information (such as programs) to be used for the operation of the information processing device 2600. The HDD 2604 is a storage device constructed from a nonvolatile storage medium.

[0265] Communication I / F 2605 is a communication interface based on a standard such as Wi-Fi (trademark) or 4G, and is a module for communicating with another device. Examples of display device 2606 include a liquid crystal display and an organic light emitting diode (OLED) display, and display device 2606 is used to display moving images, still images, characters, etc. Examples of input device 2607 include buttons, touch panels, keyboards, and pointing devices, and input device 2607 is used by a user to operate information processing device 2600. Display device 2606 and input device 2607 can be formed integrally as a touch panel.

[0266] Figure 26 The hardware configuration shown in is an example, and devices other than those shown may be added, or some of the devices shown may be omitted. In addition, some of the devices may be replaced with another device having the same function. Moreover, some of the functions may be provided by another device via a network, and may be shared and implemented by multiple devices for implementing the functions of the embodiment. For example, the HDD 2604 may be replaced with a solid-state drive (SSD) using a semiconductor element such as a flash memory, or may be replaced with a cloud storage device.

[0267] <Cell Image Analyzer>

[0268] In addition, if Figure 28 As shown in , the present disclosure provides a cell image analysis device, including: an image acquisition module 2801, which acquires a time series cell image group, which is obtained by collecting a plurality of cell images obtained by photographing cells at a plurality of consecutive different timings by any one of a bright field observation method and a phase contrast observation method in association with the timing; a region extraction module 2802, which extracts cell candidate regions from the cell images included in the time series cell image group; a region tracking module 2803, which determines whether the cell candidate regions on the plurality of cell images associated with different timings included in the time series cell image group correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timing, and acquires the determined cell candidate regions as the time series cell candidate region group; and an analysis module 2804, which analyzes information about the state of the cell based on information about the time series cell candidate region group. The analysis module uses a trained model, and the trained model is a machine learning model that is trained based on information about the time series cell candidate region group and information about the state of the cell obtained from the cell used for training, by using the information about the time series cell candidate region group as input and using the information about the state of the cell as output.

[0269] The image acquisition module 2801, the region extraction module 2802, the region tracking module 2803 and the analysis module 2804 respectively perform the image acquisition step, the region extraction step, the region tracking step and the analysis step described so far.

[0270] <Cell Image Analysis System>

[0271] In addition, if Figure 29 As shown in , the present disclosure provides a cell image analysis system including an image acquisition device 2901 and an information processing device 2902 .

[0272] The image acquisition device 2901 acquires cell images obtained by photographing cells at a plurality of consecutive different time intervals using either a bright field observation method or a phase contrast observation method. The information processing device 2902 includes: an image acquisition module 2903 that acquires cell images from the image acquisition device 2901 and acquires a time series cell image group obtained by collecting the cell images in association with the time intervals; a region extraction module 2904 that extracts cell candidate regions from the cell images included in the time series cell image group; and a region tracking module 2905 that determines the region included in the cell image. Whether the cell candidate areas on multiple cell images associated with different timings in a time series cell image group correspond to the same object, collecting the cell candidate areas determined to correspond to the same object in association with the timing, and obtaining the determined cell candidate areas as a time series cell candidate area group; and an analysis module 2906, which analyzes information about the state of the cell based on the information about the time series cell candidate area group, and the analysis module uses a trained model, which uses the information about the time series cell candidate area group as input for the cell used for training and uses the information about the state of the cell as output.

[0273] Image acquisition device 2901 can capture cells at multiple, consecutive, different time intervals using either brightfield or phase contrast observation methods. Image acquisition device 2901 can preferably include a lens, a detector, a camera, an excitation light source, and a storage device. Preferred examples of image acquisition device 2901 include a phase contrast microscope, a fluorescence microscope, and a confocal microscope. The information processing device is as described above. Image acquisition module 2903, region extraction module 2904, region tracking module 2905, and analysis module 2906 respectively perform the image acquisition, region extraction, region tracking, and analysis steps described above.

[0274] [Other embodiments]

[0275] The embodiment(s) of the present invention may also be implemented by a computer reading out and executing computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described embodiment(s) and / or a system or device including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by a computer of the system or device, for example, by reading out and executing computer-executable instructions from a storage medium to perform the functions of one or more of the above-described embodiment(s) and / or controlling one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)). TM ), one or more of flash memory devices, memory cards, etc.

[0276] The embodiments of the present invention can also be implemented by the following method, that is, software (including computer program products of computer programs / instructions) that perform the functions of the above-mentioned embodiments is provided to a system or device through a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer program / instructions.

[0277] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A cell image analysis method comprising: an image acquisition step of acquiring a time-series cell image group, the time-series cell image group being acquired by collecting a plurality of cell images acquired at a plurality of consecutive different timings by either a bright field observation method or a phase contrast observation method in association with the plurality of consecutive different timings; a region extraction step of extracting a cell candidate region from the plurality of cell images included in the time-series cell image group; a region tracking step of determining whether cell candidate regions on the plurality of cell images associated with mutually different timings included in the time-series cell image group correspond to the same object, collecting the cell candidate regions determined to correspond to the same object in association with the timings, and acquiring the determined cell candidate regions as the time-series cell candidate region group; as well as an analyzing step of analyzing information on the state of the cell based on the information on the time-series cell candidate region group, The analysis step involves using the trained model and The trained model includes a machine learning model, which is trained based on information about the time series cell candidate region group and information about the state of the cell obtained from the cell used for training, by using the information about the time series cell candidate region group as input and using the information about the state of the cell as output. 2 . The cell image analysis method according to claim 1 , wherein the trained model is configured to acquire information about the state of the cell based on fluorescence brightness feature value data including at least information about temporal changes in fluorescence brightness.

3. The cell image analysis method according to claim 2, The trained model is obtained by performing the following steps on the cells used for training: a fluorescence image acquisition step of acquiring a time-series fluorescence image group obtained by collecting fluorescence images corresponding to the plurality of cell images included in the time-series cell image group in association with timings; a fluorescence brightness characteristic value calculation step, wherein the fluorescence brightness characteristic value calculation step obtains fluorescence brightness characteristic value data of the time series cell candidate region group based on the time series fluorescence image group; a labeling processing step of assigning a label related to the state of the cell to the cell candidate region based on the fluorescence brightness characteristic value data; as well as a trained model generating step of generating the trained model by performing training using information on a state of a cell based on a label related to the state of the cell as output and information on a group of time-series cell candidate regions as input; The fluorescence brightness characteristic value data at least includes information about the temporal variation of the fluorescence brightness. 4 . The cell image analysis method according to claim 1 , wherein the information on the state of the cell includes at least information on a scalar value indicating one of a possibility or a binary value indicating whether the cell candidate region is a live cell region. 5 . The cell image analysis method according to claim 1 , wherein the information on the state of the cell includes at least information on a scalar value indicating one of a possibility or a binary value indicating whether the cell candidate region is a differentiated region.

6. The cell image analysis method according to any one of claims 1 to 5, The information about the time series cell candidate region group includes feature value data contained in the time series cell image group, The feature value data includes one or more items selected from the group consisting of: a brightness feature value of each cell candidate region in the cell candidate regions of the plurality of cell images; a morphological feature value of each cell candidate region in the cell candidate regions; Information about the temporal variation of brightness eigenvalues; and information about the temporal variations of morphological feature values, and The eigenvalue data includes either a scalar value or a vector.

7. A cell image analysis method according to any one of claims 1 to 5, wherein the information about the time series cell candidate area group includes time series feature value data, the time series feature value data is obtained by collecting feature value data of the cell candidate areas included in the time series cell image group in association with timing, the feature value data including one or more items selected from the group consisting of: a brightness feature value in each cell candidate area among the cell candidate areas of the multiple cell images; and a morphological feature value in each cell candidate area among the cell candidate areas.

8. The cell image analysis method according to any one of claims 1 to 5, wherein the information about the time-series cell candidate region group includes time-series image data, which is obtained by collecting partial cell images corresponding to the cell candidate regions on the multiple cell images included in the time-series cell image group in association with timing.

9. A cell image analysis device comprising: an image acquisition module configured to acquire a time-series cell image group, the time-series cell image group being acquired by collecting a plurality of cell images acquired at a plurality of consecutive different timings by either a bright field observation method or a phase contrast observation method in association with the plurality of consecutive different timings; a region extraction module configured to extract candidate cell regions from the plurality of cell images included in the time series cell image group; a region tracking module configured to determine whether cell candidate regions on the plurality of cell images associated with mutually different timings included in the time-series cell image group correspond to the same object, collect the cell candidate regions determined to correspond to the same object in association with the timings, and acquire the determined cell candidate regions as the time-series cell candidate region group; as well as an analysis module configured to analyze information about the state of the cell based on information about the time-series cell candidate region group, wherein the analysis module is configured to use the trained model, and The trained model includes a machine learning model, which is trained based on information about the time series cell candidate region group and information about the state of the cell obtained from the cell used for training, by using the information about the time series cell candidate region group as input and using the information about the state of the cell as output. 10 . A medium having a program stored thereon in a computer-readable format, the program for causing a computer to execute the cell image analysis method according to claim 1 .

11. A cell image analysis system comprising: Image acquisition equipment; as well as information processing equipment, The image acquisition device is configured to acquire cell images obtained by either a bright field observation method or a phase contrast observation method at a plurality of consecutive different timings, The information processing equipment includes: an image acquisition module configured to acquire cell images from an image acquisition device and acquire a time series cell image group obtained by collecting the cell images in association with the plurality of consecutive different timings; a region extraction module configured to extract candidate cell regions from the cell images included in the time series cell image group; a region tracking module configured to determine whether cell candidate regions on a plurality of cell images associated with mutually different timings included in the time-series cell image group correspond to the same object, collect the cell candidate regions determined to correspond to the same object in association with the timings, and acquire the determined cell candidate regions as the time-series cell candidate region group; and an analysis module configured to analyze information about the state of the cell based on information about the time-series cell candidate region group, and The analysis module is configured to use a trained model that uses information about a time series cell candidate region group as input and information about a state of the cell as output for cells used for training.

Images