Cell image analysis method

Abstract

To provide a cell image analysis method high in accuracy.SOLUTION: A cell image analysis method comprises: an image acquisition step of acquiring a time-sequence cell image group collecting a plurality of cell images photographing a cell at a plurality of continuous different timings in association with the timing; an area extracting step of extracting a cell candidate area from the cell image; an area tracking step of, regarding a plurality of cell candidate areas of the cell image included in the time-sequence cell image group and associated with respective different timings, collecting the cell candidate area determined to correspond to the same object in association with the timing, and acquiring the collected cell candidate areas as a time-sequence cell candidate area group; and an analysis step of analyzing information on a cell state on the basis of information on the time-sequence cell candidate area group, in which the analysis step is configured to use a learned model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a cell image analysis method, a program, a cell image analysis device, and a cell image analysis system. [Background technology]

[0002] In the field of cell culture, cell images are often analyzed to determine cell status, such as whether the cells are alive or dead. In particular, techniques that use machine learning to analyze cell images have attracted attention.

[0003] Patent Document 1 discloses a technique for non-invasively identifying live and dead cells from bright-field images. In this technique, live and dead cells are identified from fluorescent images in order to obtain training data for machine learning. Patent Document 2 discloses a technique for determining whether a cell is viable or dead based on the morphological characteristics of time-series cell images. This technique also describes the use of stained samples to serve as training data for machine learning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-85966 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-210212 Summary of the Invention [Problem to be solved by the invention]

[0005] However, because cell states, such as whether a cell is alive or dead, change over time, creating training data based only on fluorescence images taken at a specific time, as in Patent Documents 1 and 2, results in training a machine learning model using data with inaccurately labeled answers. Analysis using such a machine learning model may lack accuracy. The present disclosure addresses the need to provide a cell analysis method that creates highly reliable training data. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, in one aspect, the present disclosure provides a cell image analysis method comprising: an image acquisition step of acquiring a time-series cell image group in which a plurality of cell images of cells are captured by either bright-field observation or phase-contrast observation at a plurality of consecutive different timings, and the acquired images are associated with the timings; a region extraction step of extracting cell candidate regions from the cell images included in the time-series cell image group; a region tracking step of determining whether the cell candidate regions in a plurality of cell images included in the time-series cell image group, each associated with a different timing, correspond to the same subject, and collecting the cell candidate regions determined to correspond to the same subject in association with the timings to acquire a time-series cell candidate region group; and an analysis step of analyzing information about the state of the cells based on information about the time-series cell candidate region group, wherein the analysis step uses a trained model, and the trained model is a machine learning model that has been trained based on information about the time-series cell candidate region group and information about the state of the cells acquired from training cells, using information about the time-series cell candidate region group as input and information about the state of the cells as output. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to create more reliable training data. The training data created according to the present disclosure can be used to improve the performance of a machine learning model that distinguishes the state of a cell. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a process flow diagram of the cell image analyzing method according to the first embodiment. [Figure 2] 10 shows an example of a group of time-series phase-contrast images acquired in step S101. [Figure 3] An example of the mask image acquired in step S102 is shown. [Figure 4]An example of the area tracking result acquired in step S103 is shown below. [Figure 5] An example of the feature amount data group calculated in steps S104 and S105 will be shown below. [Figure 6] FIG. 1 is a processing flow diagram of a trained model generation method according to a first embodiment. [Figure 7] 10 shows an example of a group of time-series phase-contrast images and a fluorescent image acquired in step S601. [Figure 8] An example of the feature amount data group calculated in steps S604 to S606 will be shown below. [Figure 9] 10 shows an example of a graph of the time transition of the fluorescence intensity of each time-series cell candidate region group. [Figure 10] 10 shows an example of a fluorescence intensity feature amount data group of each time-series cell candidate region group calculated in step S606. [Figure 11] 10 shows an example of a group of fluorescence intensity feature amount data of each group of time-series cell candidate regions after labeling processing. [Figure 12] 10 shows an example of a group of feature amount data for each cell candidate region after labeling processing. [Figure 13] 10 shows an example of a graph of the time transition of the fluorescence intensity of each time-series cell candidate region group for each label. [Figure 14] An example of obtaining a truth value for each label using a trained model in step S106 is shown below. [Figure 15] An example of obtaining the likelihood for each label using a trained model in step S106 is shown below. [Figure 16] 10 shows an example of a group of time-series phase-contrast images and a fluorescence image acquired in Modification 2 of the first embodiment. [Figure 17] FIG. 11 is a data flow diagram of a step-by-step labeling process according to a third modification of the first embodiment. [Figure 18] An example of extracting live cell and dead cell data from data labeled as cells with changes in viability and death will be described below in Modification 5 of the first embodiment. [Figure 19]13 shows an example of the results of performing a cleansing process on data labeled as non-cells in the fourth modification of the first embodiment. [Figure 20] FIG. 10 is a process flow diagram of a cell image analyzing method according to a second embodiment. [Figure 21A] This is a part of a diagram explaining input and output to a trained model according to the second embodiment, and shows an example of a group of feature data. [Figure 21B] FIG. 10 is a part of a diagram illustrating input and output to a trained model according to the second embodiment, showing an example of the results of extracting time-series feature data. [Figure 21C] This is a part of a diagram explaining the input and output to a trained model according to the second embodiment, showing an example in which a true / false value result is output. [Figure 22] FIG. 10 is a processing flow diagram of a trained model generation method according to a second embodiment. [Figure 23] FIG. 10 is a process flow diagram of a cell image analyzing method according to a third embodiment. [Figure 24A] This is a part of a diagram showing an example of input data to a trained model related to the third embodiment, and shows an example of a group of partial cell images cut out from each cell image based on the area tracking number and the cell candidate area. [Figure 24B] This is a part of a diagram explaining the input and output to a trained model according to the third embodiment, showing an example in which a true / false value result is output. [Figure 25] FIG. 11 is a processing flow diagram of a trained model generation method according to the third embodiment. [Figure 26] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an information processing device capable of executing a program according to the present disclosure. [Figure 27] 10 shows an example of a group of time-series phase-contrast images and a fluorescence image acquired in Modification 3 of the first embodiment. [Figure 28] 1 shows an example of a cell image analyzer. [Figure 29] 1 shows an example of a cell image analysis system. [Figure 30] An example of the results of evaluating the accuracy of the live cell determination model is shown below. [Figure 31] The results of evaluating the accuracy rate (viable cells) for each culture time are shown. [Figure 32] The results of evaluating the accuracy rate (non-viable cells) for each culture time are shown. [Figure 33] The results of calculating the accuracy evaluation index for the incubation time period (0 to 11 hours) and the incubation time period (12 hours or more) are shown. [Figure 34] The results of evaluating the accuracy rate (viable cells) for each culture time are shown. [Figure 35] The results of evaluating the accuracy rate (non-viable cells) for each culture time are shown. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] The present disclosure includes an image acquisition step of acquiring a time-series cell image group in which a plurality of cell images of cells are captured by either bright-field observation or phase-contrast observation at a plurality of consecutive different timings and collected in association with the timings; a region extraction step of extracting cell candidate regions from the cell images included in the time-series cell image group; a region tracking step of determining whether the cell candidate regions of the plurality of cell images included in the time-series cell image group and associated with different timings correspond to the same object, collecting the cell candidate regions determined to correspond to the same object in association with the timings, and acquiring them as a time-series cell candidate region group; and an analysis step of analyzing information related to the state of the cells based on information about the time-series cell candidate region group, wherein the analysis step uses a trained model, The cell image analysis method is characterized in that the trained model is a machine learning model that has been trained based on information about time-series cell candidate region groups and information about the state of the cells obtained from training cells, using information about the time-series cell candidate region groups as input and information about the state of the cells as output.

[0011] In the trained model, information about the state of the cell may be acquired based on fluorescence brightness feature data that includes at least information about changes in fluorescence brightness over time. In this method, the trained model is obtained by carrying out the following steps: a fluorescence image acquisition step of acquiring a time-series fluorescence image group for the training cells, in which fluorescence images corresponding to the cell images included in the time-series cell image group are collected in association with the timing; a fluorescence brightness feature calculation step of acquiring fluorescence brightness feature data for the time-series cell candidate region group based on the time-series fluorescence image group; a labeling step of assigning labels related to the state of the cells to the cell candidate regions based on the fluorescence brightness feature data; and a trained model generation step of generating the trained model by training using information about the time-series cell candidate region group as input and outputting information about the state of the cells based on the labels related to the state of the cells;

[0012] The information about the state of the cell can include at least information about a true / false value indicating whether or not the cell candidate region is a live cell region, or a scalar value indicating the possibility. Furthermore, the information about the state of the cell can include at least information about a true / false value indicating whether or not the cell candidate region is a differentiated region, or a scalar value indicating the possibility.

[0013] First Embodiment (overview) In this embodiment, we will explain a method for obtaining information about the state of cells and a method for generating the trained model by inputting a group of time-series cell images, which are collected by correlating multiple cell images captured using phase-contrast observation with the timing, i.e., a feature vector composed of brightness features and morphological features calculated from the group of time-series phase-contrast images or features related to their changes over time, into a trained model. As a good example of application of this embodiment, an example will be described in which a cell sample immediately after seeding stem cells (iPS cells: induced pluripotent stem cells, ES cells: embryonic stem cells) is observed, and information indicating whether the cell is a "live cell," "dead cell," or "non-cell" is obtained as information regarding the cell state. The information regarding the cell state is not limited to these, and may be information indicating whether the cell is a "live cell" or a "dead cell," or information indicating whether the cell is "post-differentiation" or "pre-differentiation."

[0014] (Processing Procedure) FIG. 1 shows a flow diagram of a cell image analysis method according to the first embodiment.

[0015] (Step S101: Acquiring a group of time-series cell images) Step S101 is an image acquisition step of acquiring a plurality of cell images captured by bright field or phase contrast at different successive times using a cell culture observation device in association with the times. In step S101, cell image data captured by the cell culture observation device is acquired. Here, the cell images are data of time-series phase-contrast images captured at predetermined intervals for one culture vessel over a certain period of time. For example, the time captured immediately after seeding is set to 0 hour, and 52 images are acquired at 1-hour intervals up to 51 hours later. Figure 2 is a diagram showing an example of a time-series cell image group 202 acquired in a culture vessel 201. Cell images I00 to I51 correspond to images captured at each time from 0 to 51 hours of culture time, respectively.

[0016] The image acquired in this step is not limited to a phase-contrast image, but may be an image characterized by enhanced contrast of the cellular contours relative to non-cellular regions. For example, this method can be suitably applied to images such as differential interference optical systems, oblique illumination systems, optical systems in which both the object side and the image side are telecentric (double-telecentric optical systems), and defocused images (bright-field images taken at a fixed distance from the focal point in the optical axis direction). Furthermore, it is preferable that each cell image is captured so as to include a common field of view for the culture vessel. In this embodiment, it is assumed that all cell images I00 to I51 are captured from the same field of view. Note that the cell image data may be acquired in real time from the cell culture observation device, or may be acquired from an external storage area such as a hard disk drive or cloud storage. In this step, a group of time-series cell images showing the morphology of the cells photographed by the cell culture observation device is acquired, and the data is output to the execution unit of step S102 (for example, the region extraction unit 2802). Note that outputting information includes the case where the information is output after being stored in a storage device.

[0017] (Step S102: Obtaining cell candidate regions) Step S102 is a cell candidate region acquisition step for acquiring a cell candidate region in each of the cell images included in the time-series cell image group acquired in step S101. Here, the cell candidate region refers to an object region whose outline is emphasized in the phase contrast image acquired in step S101. Specifically, in addition to the cell region, it includes objects such as cell debris, scratches on the culture vessel, and adhesions. In step S102, these regions are acquired as cell candidate regions. Cell candidate regions in phase-contrast images can be obtained using known image analysis techniques, such as image processing techniques like active contouring and GraphCut, or a trained model in which a deep learning network like U-Net is trained using a mask image representing the cell candidate region as a training model. In this embodiment, an example will be described in which a cell candidate region is acquired using a differential filter and binarization processing. First, a differential image is generated by applying a differential filter to the cell image. A differential image is an image that calculates the amount of difference in brightness between each pixel and its surrounding pixels and is expressed as an image. In the case of a cell image, the differential image is an image with high brightness values ​​at the outlines of the cell candidate region and at the outlines of cells within the cell candidate region.

[0018] Next, the generated differential image is subjected to a binarization process to identify areas with high brightness values ​​in the differential image. In the binarization process, an arbitrary threshold is set, and the value of each pixel in the differential image is replaced with a value of 1 if the pixel value is equal to or greater than the threshold, and a value of 0 if the pixel value is less than the threshold. Here, the binarization method is not limited to a method of setting an arbitrary threshold. For example, a method of automatically determining a threshold, such as Otsu's binarization or Li's binarization, may also be used. When an arbitrary threshold is set, the threshold is set according to the imaging conditions of the device, such as exposure time and focus setting. Alternatively, a method of determining a threshold for each pixel of the image, such as adaptive binarization, may also be used. The binarization process creates a binary image (hereinafter referred to as an edge image) in which pixel values ​​in areas with large changes in brightness are represented as 1 and other pixel values ​​are represented as 0.

[0019] Next, a mask image of the cell candidate region is generated based on the generated edge image. Here, the mask image is a binary image in which the cell candidate region is represented by a pixel value of 1 and other regions by a pixel value of 0. The mask image is generated by extracting regions in the edge image where pixel values ​​of 1 are connected, and replacing the pixel values ​​inside each connected region with 1.

[0020] Here, if the size of the target cell can be estimated, upper and lower limits for the size of the cell candidate region may be set in this step, and cell candidate regions outside the range may be excluded. For example, in the case of stem cells, an area of ​​10,000 μm 2 Set it so that the above areas are excluded. By carrying out the above processing, cell candidate regions are obtained in each image acquired in step S101.

[0021] Figure 3 shows an example of mask images M00 to M51 obtained by applying the above-described processing to each of the cell images I00 to I51 in Figure 2. Areas of black pixels in the mask images indicate the respective cell candidate regions. As shown in Figure 3, identifiers such as C001 and C002 are assigned to each of the cell candidate regions so that it is possible to distinguish which cell image each cell candidate region belongs to. In this step, a mask image in which each cell candidate region can be identified and contour coordinate information corresponding to each cell candidate region are obtained as information indicating the cell candidate region in each cell image included in the time-series cell image group obtained in step S101, and this information is output to the execution unit of step S103 (e.g., the region tracking unit 2803).

[0022] (Step S103: Tracking cell candidate regions) Step S103 is a region tracking step in which, for the cell candidate regions acquired in step S102, it is determined whether the cell candidate regions in multiple cell images associated with different timings are the same region (corresponding to the same object), and the cell candidate regions determined to correspond to the same object are collected in association with the timing and acquired as a time-series cell candidate region group. For a cell candidate region acquired at a specific timing, that is, acquired in a cell image associated with a specific timing, which cell candidate region at the next timing is the same cell candidate region can be determined by calculating the degree of overlap of the regions. For example, the degree of overlap m of two regions A1 and A2 is calculated using the following equation 1.

number

[0023] Hereinafter, an example will be described in which the cell candidate regions in FIG. 4 are used to determine which cell candidate regions between consecutive cell images are the same region. First, the degree of overlap m between the cell candidate region C001 in cell image M00 and each cell candidate region in cell image M01 is calculated, and the cell candidate region in cell image M01 with the largest degree of overlap is deemed to correspond to the same object as cell candidate region C001, and in such a case, these cell regions are said to be the same. In the example of Figure 4, cell candidate region C001 and cell candidate region C011 are determined to be the same region. Next, for the next cell candidate region C002, the degree of overlap m with each cell candidate region in the cell image I01 is calculated in the same way as for cell candidate region C001, and the cell candidate region with the greatest degree of overlap is determined to be the same cell region as cell candidate region C002. At this time, cell candidate region C011, which has already been determined to be the same as cell candidate region C001, is excluded from the target.

[0024] Thereafter, by repeatedly performing the same process on other cell candidate regions and other cell images, it is determined which cell candidate regions are the same region between cell images associated with different timings. In cell culture, regions corresponding to detached adhered cells and floating debris are also included as cell candidate regions. In such cases, it is possible that a cell candidate region identical to a cell candidate region at one timing may not exist at the next timing. Therefore, it is effective to set a threshold value for the degree of overlap and determine that a cell candidate region with the largest degree of overlap is the same cell candidate region when the degree of overlap is equal to or greater than the threshold value. The threshold value can be set to, for example, 0.25.

[0025] Regions determined to be identical in this step are assigned identifiers that allow identification of the identical cell candidate regions. Figure 4 shows an example of some of the identifiers assigned as a result of applying the processing of this step to the cell candidate regions of Figure 3. Cell candidate regions C001, C011, C121, and C511 and cell candidate regions C002, C012, C122, and C512 are determined to be identical regions, and are assigned identifiers T001 and T002, respectively. Hereinafter, identifiers T001 and T002 will be assigned to cell candidate regions corresponding to the same target at different times, as shown in Figure 5, and these will be referred to as region tracking numbers. Furthermore, a group of cell candidate regions associated with different times and assigned the same region tracking number will be referred to as a time-series cell candidate region group.

[0026] As described above, in this step, for the cell candidate regions acquired in step S102, region tracking numbers indicating which cell candidate regions are the same cell candidate regions at different timings are assigned to the mask image and the contour coordinate information corresponding to each cell candidate region. This information is output to the execution unit in step S104.

[0027] While the example in which the degree of overlap based on Equation 1 is calculated and whether or not the regions are the same is shown as a preferred example, the method is not limited to Equation 1, and the determination may be made using a known evaluation index relating to the degree of overlap between regions, such as DICE or IoU, or the movement distance of the center of gravity of the regions, etc. Also, it is possible to apply a known region tracking method, such as a method for calculating optical flow or a Kalman filter.

[0028] (Step S104: Calculation of first feature amount) Step S104 is a first feature amount calculation step of calculating brightness features or morphological features, or both, of each of the cell candidate regions included in the time-series cell candidate region group acquired up to step S103. The brightness feature is a scalar value calculated as a statistical quantity of brightness values ​​in a cell candidate region, independent of time change, and the statistical quantity is, for example, the average value, minimum value, or deviation. The morphological feature is a scalar value that quantifies the size or shape of a cell candidate region, and for example, the area or diameter is calculated. The brightness feature and morphological feature are calculated for each cell candidate region acquired up to step S103. The calculated feature amount is linked to the identification number of the cell candidate region so that it can be identified which cell candidate region the feature amount belongs to. In this step, a data group of brightness feature amounts and morphological feature amounts for each cell candidate region acquired up to step S103 is calculated and acquired, and this information is output to the execution unit in step S105. The calculated feature amounts are merely examples, and for example, the maximum value or median value may be calculated as the brightness feature amount, and the circularity or area envelopment ratio may be calculated as the morphological feature amount.

[0029] (Step S105: Calculation of second feature amount) Step S105 is a second feature calculation step in which the first feature acquired in step S104 or a feature indicating a temporal change in the position information (time-series feature) is calculated based on each time-series cell candidate region group acquired in step S103.

[0030] The second feature is obtained, for example, by extracting a set of data on the first feature and position information of the same cell candidate region at a given timing within a certain period of time in the past, and calculating the change amount based on linear approximation or the average difference between temporally consecutive data. For example, the change amount of diameter based on linear approximation and the average movement amount of the center of gravity coordinate are calculated based on data from the past 12 hours. Specifically, an example of calculating the change amount of diameter and the average movement amount of the center of gravity for the cell candidate region C121 in the example of Figure 4 will be described.

[0031] First, for cell candidate region C121, diameter data and centroid coordinates are obtained for the past 12 hours up to C011, which is the same cell candidate region. Next, the diameter change amount is calculated based on the obtained diameter data for 12 hours. The diameter change amount can be determined by treating the diameter data for 12 hours as a data group consisting of two variables, time and diameter, and finding the slope when linear approximation is performed using the least squares method. Furthermore, the average movement amount is calculated based on the obtained centroid coordinate data for 12 hours. The average movement amount can be determined by calculating the Euclidean distance between temporally consecutive centroid coordinate data and averaging them. By repeating the above process for other time-series cell candidate region groups, a second feature data group for each time-series cell candidate region group is obtained.

[0032] Table 500 in FIG. 5 is an example of the results of calculating the first feature amount data and the second feature amount data for each cell candidate region in FIG.

[0033] In this step, feature data including the first feature or the second feature, which is a feature indicating a change in location information over time, is calculated, and this information is output to the execution unit (e.g., the analysis unit 2804) in step S106.

[0034] While the preferred example has been described in which the second feature amount is calculated based on data from the past 12 hours, the present invention is not limited to this example and the second feature amount may be calculated based on data from the past 4 hours, 6 hours, etc. Furthermore, the example has been described in which the amount of change in diameter or the average amount of movement is calculated, but the present invention is not limited to this example and the amount of change in brightness feature amount may be calculated, or the variance of data may be calculated instead of the amount of change based on linear approximation or the average value of the difference.

[0035] (Step S106: Analysis process) Step S106 is an analysis step, in which information about the state of the cells in the cell candidate region is obtained using the feature amounts calculated in steps S105 and S104. Here, the information about the state of the cells in the cell candidate region is, for example, a true / false value indicating whether the cell candidate region is in a live cell, dead cell, or non-cellular state.

[0036] In this step, a learned model that has been pre-trained to input a feature vector composed of the first feature from step S104 and the second feature from step S105 and output information about the state of cells in the cell candidate region is used. Here, the trained model is a machine learning model that has been trained using as training data the results of a labeling process based on information about fluorescence brightness, including at least information about changes in fluorescence brightness over time. Below, we will explain the specific method for generating a trained model.

[0037] (Processing procedure for generating trained models) 6 is a flow diagram showing the flow of generating a trained model used in the analysis step of the first embodiment. The following steps are performed on the training cells.

[0038] (Step S601: Obtain a time-series cell image group) Step S601 is an image acquisition process for acquiring a group of time-series cell images to be used for training the learned model.

[0039] In the above-mentioned step S101, a time-series cell image group was obtained using phase-contrast images taken at multiple different consecutive timings as cell images, but in this step, in addition to these, fluorescent images of a cell sample, which is a learning cell corresponding to each timing, are also obtained. The phase-contrast images and fluorescent images collected in association with timing may be referred to as a time-series phase-contrast image group and a time-series fluorescent image group, respectively.

[0040] To generate a trained model that outputs a true / false value indicating whether a cell candidate region is in a live, dead, or non-cellular state, a phase-contrast image is obtained by phase-contrast observation of a cell sample to which a reagent that exhibits fluorescent properties in live and dead cells has been added, and a fluorescent image is obtained by fluorescent observation. For example, a reagent that exhibits fluorescent properties in live cells and a reagent that exhibits fluorescent properties in dead cells can be used, each with different fluorescent properties, to distinguish between live and dead cells. Alternatively, when both live and dead cells are stained with fluorescent dyes of the same color, images can be obtained by photographing cell samples to which staining reagents for dead cells and live cells have been added at different times.

[0041] 7 shows an example of image group 702, in which cells containing the same color fluorescent dye were first seeded, and a reagent for fluorescently staining dead cells was added, followed by photographing the cells for 48 hours, and then a reagent for fluorescently staining live cells was added, and phase-contrast and fluorescent images of the cells were acquired for 51 hours. The image taken immediately after seeding is the image at 0 hours of culture time, and phase-contrast images I700 to I751 and fluorescent images FL700 to FL751 were acquired at one-hour intervals for up to 51 hours. In this step, a group of time-series phase contrast images and a group of time-series fluorescence images of the cell sample corresponding to each time are acquired, and this information is output to the execution unit of step S602.

[0042] (Steps S602 to S605) The processing in steps S602 to S605 is the same as applying the processing in steps S102 to S105 to the time-series phase-contrast image group acquired in step S601, and therefore a description thereof will be omitted.

[0043] Through the processing of steps S602 to S605, a group of feature amount data as shown in FIG. 5 is calculated from the group of time-series phase difference images acquired in step S601, and this information is output to the execution section in step S606.

[0044] (Step S606) Step S606 is a fluorescence intensity feature calculation step for calculating a feature amount related to fluorescence intensity in the time-series cell candidate region group based on the fluorescence image acquired in step S601 and each cell candidate region and region tracking number acquired in steps S602 to S605.

[0045] First, the average value of the fluorescence intensity in each cell candidate region is calculated for the fluorescence image acquired in step S601. If the fluorescence image is an RGB color image, calculation is performed for the intensity value of the color component corresponding to the fluorescent dye used to stain the cell sample. As explained in step S601, if the sample is stained with a green fluorescent dye, the average value of the intensity of the G component is calculated.

[0046] Figure 8 shows a table 801 as an example of calculated average values ​​of fluorescence brightness in each cell candidate region. Average fluorescence brightness data is added to the feature data group calculated from the phase contrast image as in Figure 6. Hereinafter, the average fluorescence brightness will be referred to simply as fluorescence brightness. Here, when using a cell sample in which dead and live cells are stained with the same fluorescent dye at different times, as in the example of Figure 7, the fluorescence intensity calculated from fluorescence images FL749-FL750 is the sum of the fluorescence intensity due to the staining of live cells and the fluorescence intensity due to the staining of dead cells. Therefore, it is preferable to subtract the fluorescence intensity due to the staining of dead cells from the fluorescence intensity calculated from the fluorescence images after live cell staining to obtain the fluorescence intensity of live cells. For example, in the example of Figure 7, the fluorescence intensity of each cell candidate region calculated in fluorescence images FL749-FL751 at 49 hours or later can be subtracted from the fluorescence intensity of the same cell candidate region calculated in fluorescence image FL748 at 48 hours. Figure 9 shows a graph 901 of the time progression of fluorescence intensity in each time-series cell candidate region group after the above-mentioned subtraction process. The horizontal axis represents the number of days in culture, the vertical axis represents fluorescence intensity, and each line represents each time-series cell candidate region group.

[0047] Next, the fluorescence intensity feature of each time-series cell candidate region group is calculated using the fluorescence intensity of each cell candidate region calculated as shown in Table 800 in Figure 8. Here, the fluorescence intensity feature is a feature used to label each time-series cell candidate region group with a label indicating the cell state of the cell candidate region, such as "live cell," "non-cell," "dead cell," or "cell with a change in viability / death status," in the labeling process described below. Therefore, it is desirable that the fluorescence intensity feature be a feature useful for distinguishing between them. For example, four scalar values ​​are calculated for the cell candidate regions of the time-series cell candidate region group: "fluorescence intensity at the beginning of dead cell staining," "fluorescence intensity at the end of dead cell staining," "maximum change in fluorescence intensity of dead cell staining," and "maximum fluorescence intensity of live cell staining." For "fluorescence intensity at the beginning of dead cell staining" and "fluorescence intensity at the end of dead cell staining," the fluorescence intensity data of the cell candidate region of the cell image at the corresponding timing may be extracted depending on the timing of staining the live cells. For example, in the example of Figure 7, the fluorescence intensity at 0 and 48 hours, respectively, is extracted. For the "maximum amount of change in fluorescence brightness of dead cell staining," the amount of change in fluorescence brightness can be determined by linearly approximating the group of fluorescence brightness data for the past 12 hours from any timing, as in the method described in step S105, and using the slope as the amount of change. The amount of change in fluorescence brightness at each timing in the group of time-series cell candidate regions is calculated, and the maximum value of these is obtained. For the "maximum amount of fluorescence brightness of live cell staining," the maximum value can be obtained from the fluorescence brightness data after the timing at which the live cells were stained.

[0048] Figure 10 shows a table 1000 as an example of calculation of the fluorescence intensity feature of each time-series cell candidate region group from the data of each cell candidate region including fluorescence intensity as shown in Figure 8. The fluorescence intensity feature is calculated for each time-series cell candidate region group having the same tracking number. As described above, in this step, the feature amount related to the fluorescence intensity for each cell candidate region in the time-series cell candidate region group is calculated, and this information is output to the execution section of step S607.

[0049] Although an example in which four scalar values ​​are calculated as the fluorescence intensity feature has been described, the scalar values ​​calculated as the fluorescence intensity feature are not limited to these. For example, statistical quantities such as the mean value, deviation, minimum value, and maximum value of the fluorescence intensity of the dead cell stain in each time-series cell candidate region group may be calculated.

[0050] (Step S607) Step S607 is a labeling process step in which information about the state of the cell is assigned to each time-series cell candidate region group based on the fluorescence intensity feature calculated in step S606. Here, the information about the state of the cell is a correct answer label that serves as a teacher for training a trained model, which will be described later.

[0051] First, an unsupervised clustering method is used to divide the fluorescence intensity feature data group, such as that shown in FIG. 10, acquired in step S606 into data groups of arbitrary clusters. Here, the time-series cell candidate region group can be assumed to be one of four types: "live cells," "non-cells," "dead cells," and "cells undergoing a change in viability." Therefore, the arbitrary number of clusters can be set to four, and a known unsupervised clustering method can be applied. For example, hierarchical clustering can be applied. However, other methods such as K-means can also be applied.

[0052] Next, each cluster divided by unsupervised clustering is labeled as a "live cell," "non-cell," "dead cell," or "cell with a change in viability." This labeling process can be performed rule-based based on the average value of the fluorescence intensity feature data in each cluster. First, if the time-series cell candidate region group is "dead cell," it is expected that both the "fluorescence intensity at the beginning of dead cell staining" and the "fluorescence intensity at the end of dead cell staining" will be large. Therefore, the average values ​​of the "fluorescence intensity at the beginning of dead cell staining" and the "fluorescence intensity at the end of dead cell staining" among the fluorescence intensity feature data group belonging to the cluster are calculated, and the cluster with the largest average value is designated as "dead cell." Next, if the time-series cell candidate region group is "cell with a change in viability," it is expected that there is a temporal change in the fluorescence intensity of the dead cell stain. Therefore, the average value of the "maximum change in fluorescence intensity of dead cell staining" among the fluorescence intensity feature data group belonging to the cluster is calculated, and the cluster with the largest average value is designated as "cell with a change in viability." Furthermore, if the time-series cell candidate region group is "living cells," it is likely that the "fluorescence intensity of living cell staining" will be a large value. Therefore, the average value of the "maximum fluorescence intensity of living cell staining" is calculated from the data group of fluorescence intensity features belonging to the cluster, and the cluster with the largest average value is designated as "living cells." Finally, the remaining clusters are designated as "non-cells."

[0053] The rule-based method for labeling each cluster is not limited to the method described above. For example, instead of labeling in the order of "dead cells," "cells in a state of change in viability," "live cells," and "non-cells," the order of labeling may be "cells in a state of change in viability," "dead cells," "live cells," and "non-cells." Furthermore, it is expected that "non-cells" have both low "staining intensity of live cells" and low "staining intensity of live cells." Therefore, the method may include a process of calculating the average values ​​of the "final fluorescence intensity of dead cell staining" and the "maximum fluorescence intensity of live cell staining" among the data sets of fluorescence intensity features belonging to a cluster, and labeling the cluster with the smallest average value as "non-cell."

[0054] Through the above process, it is determined whether each cluster corresponds to a "live cell," "non-cell," "dead cell," or "cell with a change in viability," and a corresponding label is assigned to each time-series cell candidate region group belonging to each cluster as information about the cell state. Furthermore, by referring to the region tracking number and label in Figure 11, it is also possible to assign a label to each cell candidate region in the feature amount data group of each cell candidate region as shown in Figure 8.

[0055] Figure 11 is a table showing the results of applying the labeling process described above to the fluorescence intensity feature data group of the time-series cell candidate region group in Figure 10. The label column in Table 1100 indicates whether the cell is a "live cell," "non-cell," "dead cell," or "cell with a life-or-death transition," and the labels are converted to scalar values ​​of 0, 1, 2, or 3. Table 1200 in Figure 12 also shows the results of labeling each cell candidate region in Figure 8 using the region numbers and labels in Figure 11. Figure 13 is a graph showing the time progression of fluorescence intensity for each time-series cell candidate region group in Figure 9, divided and plotted based on the results of the labeling process. Graphs 1301, 1302, 1303, and 1304 show the time progression of fluorescence intensity for each time-series cell candidate region group labeled as a "live cell," "dead cell," "cell with a life-or-death transition," and "non-cell," respectively. As described above, in this step, a group of feature amount data for each cell candidate region to which information on the state of the cell has been added is output to the execution section of step S608.

[0056] (Step S608: Generate a trained model) Step S608 is a trained model generation step for generating a trained model based on the feature amount data group of each cell candidate region to which information regarding the state of the cell obtained in step S607 has been added.

[0057] In this step, a trained model is generated by training a machine learning model that inputs a feature vector composed of each feature calculated from the phase contrast image and outputs a true / false value indicating whether the cell is alive, dead, or non-cellular, using the label assigned in step S607 as training material. In the example of FIG. 12, the feature values ​​calculated from the phase contrast image are diameter, average brightness, minimum brightness, brightness deviation, average movement, and diameter change. The trained model is a model structure and parameters optimized based on training data so that a feature vector is input and a true / false value indicating the correct data label is obtained. In the data obtained as shown in Figure 12, data labeled with label 2 are "cells that change between live and dead" when observing the same cells over time, and are likely to contain ambiguous states where it is unclear whether the cell is live or dead. Data containing such ambiguous states reduces the performance of machine learning models that classify live and dead cells. For this reason, data excluding label 2 data is used as training data to generate a trained model.

[0058] The machine learning model may be any model that inputs a feature vector and classifies it into an arbitrary number of classes, and a method such as XGBoost is used. Without being limited to this, other models such as neural networks, SVMs (Support Vector Machines), random forests, LightGBMs, and Gaussian mixture models can also be applied. Furthermore, the structure and parameters of these models can be optimized based on the training data using known optimization methods. For example, in the case of XGBoost, optimization is performed using methods such as grid search and Bayesian optimization. Note that an anomaly detection-based model can also be used when performing two-class classification, such as "live cell" and "other" states, rather than "live cell," "dead cell," and "non-cell" states. For example, a model that fits to a probability density distribution, such as kernel density estimation, can be used. Anomaly detection-based models are suitable for two-class classification when there is an extremely large amount of training data for the other class.

[0059] The method for generating a trained model in the analysis step of the first embodiment has been described above. In step S601, an example of acquiring an image acquired in an arbitrary culture vessel 701 as shown in Fig. 7 is shown, but if it is possible to acquire images of samples stained and cell images taken in a similar procedure or images of different fields of view, it is also preferable to use these to generate a trained model. In this case, the processes of steps S601 to S607 are repeatedly performed on data of different samples or different fields of view, and the data obtained in each step, such as that shown in Fig. 12, is compiled as a single training data set, and then a trained model is generated.

[0060] (Step S106: Analysis process) In step S106, the feature vectors of each cell candidate region, as shown in FIG. 5, are input to the trained model generated in steps S601 to S608 to obtain information about the state of the cells in the cell candidate region. Specifically, in the case of the trained model based on the example shown in steps S601 to S608, a true / false value indicating whether the cell is in a "live cell," "dead cell," or "non-cell" state can be obtained as an output result. FIG. 14 shows the true / false values ​​obtained using the feature vectors of each cell candidate region in FIG. 5 as input. "Live cell," "Dead cell," and "Non-cell" correspond to labels 0, 1, and 3, respectively. For example, for the cell candidate region C121, the true / false value corresponding to label 0 is 1, and the determination result is that C121 is in a "live cell" state. Note that instead of true / false values, likelihoods calculated during the process of the machine learning model outputting true / false values ​​may be obtained. FIG. 15 shows an example in which likelihoods are obtained instead of true / false values.

[0061] As described above, in step S106, the feature vector composed of the first feature from step S104 and the second feature from step S105 is input to the trained model, and information about the state of the cells in the cell candidate region is acquired.

[0062] (output of results) The cell image analysis method may further include a display control step of displaying on a display table information, such as Table 1400 shown in FIG. 14 or Table 1500 shown in FIG. 15, including the output results of the trained model for each cell candidate region, or a graph of the time progression of the truth value and likelihood for each time-series cell candidate region group. The output results of the trained model may also be displayed in the form of time-lapse images depicting the cell candidate region, a time-lapse video generated based on the time-lapse images, or a heat map image, in addition to the table information shown in FIGS. 14 and 15. For example, time-lapse images depicting the cell contours of the cell candidate region in phase-contrast images at each timing are displayed, with the display mode changed according to the trained model's judgment results. Changing the display mode here simply means that the display mode should be different for each judgment result, to the extent that the trained model's judgment results can be understood. Examples of changing the display mode include coloring the contours, changing the thickness of the contour line, changing the type of contour line (e.g., solid line, dashed line, dot-dash line, etc.), or a combination of these. Instead of displaying the cell outline, the cell area may be indicated by an arrow, the cell area may be colored, or the major and minor axes of the cell area may be indicated. Furthermore, the judgment result may be made clear by changing the brightness of the outline, color, or arrow. A heat map image may also be displayed, with the width representing time, the height representing the area tracking number, and the numerical value of each pixel representing a truth value or likelihood. In the case of a trained model that judges the states of "live cells," "dead cells," and "non-cells" as shown in the example, it is also preferable to calculate the viability (number of live cells / (number of live cells + number of dead cells)) at each timing and display the time progression of the viability (number of live cells)). As described above, this embodiment makes it possible to present to the user the judgment results regarding the state of cells using a trained model based on more reliable training data.

[0063] (Modification 1 of the First Embodiment: Alignment) In the first embodiment, an example was described in which cell images I00 to I51 were captured in the same field of view. When the cell regions are not captured in the same field of view but in the images captured in a partially shared field of view, or when blurring of the capture position due to device vibration or the like is expected, it is desirable to perform registration between multiple images in the time-series cell image group. Registration may be performed in steps S101, S102, S103, or the like, or in a separate step. Registration can be performed between images captured at successive timings, and known registration techniques such as template matching can be used. Instead of template matching, registration techniques such as SHIFT or SURF can be used, which extract feature points and estimate an affine transformation matrix based on feature point matching. Furthermore, registration may be performed using a mask image such as that shown in FIG. 3, rather than using phase-contrast images.

[0064] According to this modified example, even when there is a misalignment in the field of view between cell images at different times in a time-series cell image group, it is possible to generate a trained model and obtain information about the state of the cell using the trained model.

[0065] (Modification 2 of the First Embodiment: In the Case of a Sample in Which Live Cells and Dead Cells Are Stained with Different Dyes) In the first embodiment, in the example of the trained model generation method in steps S601 to S608, an example has been described in which fluorescent images of a cell sample in which live cells and dead cells are stained with fluorescent dyes of the same color at different times are acquired. However, the trained model generation is not limited to this, and fluorescent images of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors, i.e., with fluorescent dyes that have different excitation wavelengths and / or fluorescent wavelengths, may also be used.

[0066] In this case, in the example of Figure 7, fluorescent images corresponding to live cells and dead cells are acquired at each timing. The fluorescent images are two types of grayscale images corresponding to live cells and dead cells, respectively. If the fluorescent image is an RGB color image, the color components corresponding to the fluorescent dye used to stain the cell sample are extracted. For example, if the cell sample is stained with a red dye for live cells and a green dye for dead cells, images of the R and G components are extracted from the RGB color image. Figure 16 shows an example (image 1602) of a phase-contrast image of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors, in which a fluorescent image corresponding to dead cell staining and a fluorescent image corresponding to live cell staining are acquired. Corresponding to the phase-contrast images I1600 to I1651 at each timing, fluorescent images FL1600a to FL1651a corresponding to dead cell staining and fluorescent images FL1600b to FL1651b corresponding to live cell staining are acquired.

[0067] When fluorescent images corresponding to live cell staining and dead cell staining in the culture vessel 1601 are acquired as shown in 1601 in FIG. 16 , in step S606, the average brightness values ​​for each cell candidate region in each fluorescent image are calculated, and based on the respective fluorescent brightnesses, fluorescent brightness feature values ​​for each time-series cell candidate region group are calculated as shown in FIG. 11 . In the first embodiment, as shown in FIG. 11 , the “fluorescence brightness at the beginning of dead cell staining,” “fluorescence brightness at the end of dead cell staining,” and “maximum change in fluorescent brightness of dead cell staining” are calculated as feature values ​​representing temporal changes in the dead cell staining. In contrast, when fluorescent images are acquired for live cell staining from time 0 of the culture time, as in this modified example, it is also preferable to calculate feature values ​​representing temporal changes in the live cell staining. Specifically, as with the dead cell staining, the “fluorescence brightness at the beginning of live cell staining,” “fluorescence brightness at the end of live cell staining,” and “minimum change in fluorescent brightness of live cell staining” may be calculated.

[0068] Thereafter, in the processing from step S607 onwards, similar to the first embodiment, labeling processing is performed using the calculated fluorescence brightness feature amount to generate a trained model.

[0069] As described above, according to this modified example, even when acquiring fluorescent images of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors, it is possible to generate a trained model and obtain information about the state of cells using the trained model.

[0070] (Modification 3 of the First Embodiment: In the Case of a Sample in Which Cells and Dead Cells Are Stained with Different Dyes) In the first embodiment, in the example of the trained model generation method in steps S601 to S608, an example has been described in which fluorescent images of a cell sample are acquired in which live cells and dead cells are stained with fluorescent dyes of the same color at different times. This is not limiting, and fluorescent images of a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors may also be used in step S601. Reagents for staining cells include nuclear staining reagents and cell membrane staining reagents.

[0071] In this case, in the example of Figure 7, fluorescence images corresponding to cells and dead cells are acquired at each timing. The fluorescence images are two types of grayscale images corresponding to cells and dead cells, respectively. If the fluorescence image is an RGB color image, the color components corresponding to the fluorescent dye used to stain the cell sample are extracted. For example, if the cell sample is stained with red dye for cells and green dye for dead cells, images of the R and G components are extracted from the RGB color image. Figure 27 shows an example of acquiring phase-contrast images, fluorescence images corresponding to dead cell staining, and fluorescence images corresponding to live cell staining for a cell sample in which live cells and dead cells are stained with fluorescent dyes of different colors. Corresponding to phase-contrast images I2700 to I2751 at each timing, fluorescence images FL2700a to FL2751a corresponding to dead cell staining and fluorescence images FL2700b to FL2751b corresponding to cell staining are acquired.

[0072] When fluorescence images corresponding to cell staining and dead cell staining of culture vessel 2701 are acquired as shown in 2702 in Figure 27, in step S606, the average brightness value of each cell candidate region in each fluorescence image is calculated, and based on the respective fluorescence brightness, fluorescence brightness feature values ​​for each time-series cell candidate region group are calculated as shown in Figure 11. Furthermore, as shown in Figure 11, "fluorescence brightness at the beginning of dead cell staining," "fluorescence brightness at the end of dead cell staining," and "maximum amount of change in fluorescence brightness of dead cell staining" are calculated as feature values ​​that express temporal changes in dead cell staining. Because time changes are not large for cell staining, it is advisable to take the fluorescence brightness at a specific time point or the average value of the time-series data of fluorescence brightness.

[0073] Thereafter, in the processing from step S607 onwards, similar to the first embodiment, labeling processing is performed using the calculated fluorescence brightness feature amount to generate a trained model.

[0074] As described above, according to this modified example, even when acquiring fluorescent images of a cell sample in which cells and dead cells are stained with fluorescent dyes of different colors, it is possible to generate a trained model and obtain information about the state of cells using the trained model.

[0075] It is also preferable to use fluorescent images of cell samples in which living cells are stained with fluorescent dyes of different colors, and a similar technique can be used to generate a trained model and obtain information about the state of cells using the trained model.

[0076] (Modification 3 of the First Embodiment: Performing Unsupervised Clustering in a Stepwise Manner) In the first embodiment, a labeling method was described in which the fluorescence intensity feature data group for each time-series cell candidate region group is divided into four types of clusters by unsupervised clustering, and each cluster is labeled in a rule-based manner. Alternatively, the labeling process may be performed in a stepwise manner in step S607. This stepwise labeling process is effective when the fluorescence intensity feature values ​​that are particularly effective for classification by unsupervised clustering and the difficulty of classification are easy to predict. Specifically, in an example in which four types of labels are assigned—“live cells,” “non-cells,” “dead cells,” and “cells undergoing life-death transition”—based on the time course of fluorescence intensity as shown in FIG. 9 , it can be assumed that classification between “live cells” and “non-cells” is difficult, while classification between “dead cells” and “cells undergoing life-death transition” and the rest is relatively easy.

[0077] 17 shows a data flow when a labeling process is applied in stages to a group of fluorescence intensity feature data. As in the first embodiment, this shows an example in which four types of labels, "live cell," "non-cell," "dead cell," and "cell with change in life / death status," are assigned to a group of time-series cell candidate regions.

[0078] First, D1700 is a fluorescence brightness feature data group before labeling processing is performed, and corresponds to a data group such as the table in FIG.

[0079] Step S1701 is a first labeling process that divides the fluorescence intensity feature data set of D1700 into three data groups: "live cells / non-cells," "dead cells," and "cells with a change in viability." In this first labeling process, an unsupervised clustering method is applied with three clusters, and each cluster is labeled rule-based. Here, in the unsupervised clustering in the first labeling process, it is easy to assume that three of the fluorescence intensity feature values ​​shown in FIG. 10—"initial fluorescence intensity of dead cell staining," "final fluorescence intensity of dead cell staining," and "maximum change in fluorescence intensity of dead cell staining"—will significantly contribute to classification. Therefore, it is also preferable to perform unsupervised clustering using only these three feature values. Rule-based labeling is performed in the same manner as in the first embodiment, where "dead cells" and "cells with a change in viability" are assigned, and the remaining cluster is labeled "live cells / non-cells." By the above-described first labeling process step, the fluorescence intensity feature data group D1700 is divided into three types of data groups D1701a, D1701b, and D1701c: "live cells / non-cells," "dead cells," and "cells with changes in viability."

[0080] Step S1702 is a second labeling process that divides the data group D1701a labeled "live cells / non-cells" into two data groups: "live cells" and "non-cells." In the first labeling process, an unsupervised clustering method is applied with two clusters, and each cluster is labeled rule-based. In the unsupervised clustering in the second labeling process, it is easy to assume that the "maximum fluorescent intensity of live cell staining" feature, one of the fluorescence intensity features shown in FIG. 10, contributes significantly to classification. Therefore, it is also preferable to perform unsupervised clustering using only this one feature. Rule-based labeling is performed in the same manner as the method for assigning the "live cells" label shown in the first embodiment, and the remaining cluster is labeled "non-cells." Through the second labeling process, the fluorescence intensity feature data group D1701a is divided into two data groups D1702a and D1702b: "live cells" and "non-cells."

[0081] Finally, the data groups D1701b, D1701c, D1702a, and D1702b with determined labels are combined as labeled fluorescence brightness feature data group D1701, and a data group with correct labels can be obtained, as shown in Figures 11 and 12 in the first embodiment. The processing from step S608 onwards is the same as in the first embodiment.

[0082] As described above, according to this modified example, when the fluorescence brightness features that are particularly effective for classification by unsupervised clustering and the difficulty of classification are easy to predict, it is possible to generate a trained model using training data with a higher accuracy of correct labels.

[0083] (Modification 4 of the First Embodiment: Data Cleansing of "Cells with Changes in Life and Death") In step S608 of the first embodiment, an example was described in which, in the process of generating a trained model that classifies "live cells," "dead cells," and "non-cells," data labeled as "cells with changes in life / death" is excluded from the training data for training.

[0084] Alternatively, when generating a trained model, some cell candidate regions in the data labeled as "cells undergoing a change in viability" may be relabeled as "live cells" or "dead cells" and used as training data. Specifically, based on the amount of change in fluorescence intensity in the time-series cell candidate region group labeled as "cells undergoing a change in viability," the culture period in which the cells are in a viable state, the culture period in which the cells are transitioning from a live state to a dead state, and the culture period in which the cells are in a dead state are determined. Figure 18 shows an example of the time progression of fluorescence intensity and the time progression of change in fluorescence intensity for a time-series cell candidate region group labeled as "cells undergoing a change in viability." Graph 1800 shows the time progression of fluorescence intensity, and graph 1801 shows the time progression of change in fluorescence intensity. The time progression of fluorescence intensity for "cells undergoing a change in viability" significantly increases from a low brightness state as the cells transition from a live state to a dead state. Therefore, the timing at which the amount of change in fluorescence brightness over time reaches its maximum value is taken as the reference timing at which the cell state transitions from live to dead. The time period before the reference timing in which the amount of change is equal to or less than a threshold is taken as the time period in which the cell state is "live," and the time period after the reference timing in which the amount of change is equal to or less than the threshold is taken as the time period in which the cell state is "dead." The threshold is set to, for example, 5. In the example of FIG. 18, time period 1801a is the time period in which the cell state is "live," and time period 1801c is the time period in which the cell state is "dead." The labels in the data for the corresponding cell candidate regions are replaced with the labels "live" or "dead," respectively. Note that instead of setting a threshold, the time periods may be determined using the half-width from the reference position.

[0085] According to this modification, a trained model can be generated using training data with an increased number of training data items related to "living cells" and "dead cells."

[0086] (Modification 5 of the First Embodiment: "Non-Cellular" Data Cleansing) In the first embodiment, a labeling process was described in which labels such as "live cell," "non-cell," "dead cell," and "cell with change in viability" were assigned. In this modification, a data cleansing method will be described in which data in an even more ambiguous state is excluded from the results of the labeling process.

[0087] As a specific example, a method of data cleansing for data corresponding to the label "non-cell" in the first embodiment will be described below. As mentioned above, "non-cells" are assumed to refer to objects other than cellular regions, such as cell debris, scratches on the culture vessel, and other deposits. It is desirable for the training data for machine-learned models to be labeled with correct answers that distinguish between "cell debris," "scratches, and deposits," etc. For example, because "cell debris" is expected to take on more diverse forms than "scratches and deposits," uniformly training the model as "non-cells" may lead to a decrease in the accuracy of the machine-learned model. Therefore, a labeling process that distinguishes between "cell debris" and "scratches and deposits" from data labeled as "non-cells" may be further performed. Specifically, unsupervised clustering and rule-based labeling are performed using specific fluorescence intensity features, as shown in Variation 3 of the first embodiment. For example, unsupervised clustering is performed using the "fluorescence intensity at the end of dead cell staining" among the fluorescence intensity features shown in Figure 10, with the number of clusters set to two. The average values ​​of "fluorescence intensity at the end of dead cell staining" for each cluster are then compared, and data belonging to the cluster with the smaller average value are labeled as "non-cells," while data belonging to the other cluster are labeled as "cell debris." Figure 19 shows the results of applying data cleansing according to this modified example to a data group of a time-series cell candidate region group labeled as "non-cell" as shown in graph 1304 in Figure 13. Graph 1901 shows the time transition of the fluorescence intensity of the time-series cell candidate region group labeled as "non-cell," and graph 1902 shows the time transition of the fluorescence intensity of the time-series cell candidate region group labeled as "cell debris." In the trained model generation process in step S608, the machine learning model is trained using training data that excludes data labeled as "cell debris."

[0088] This modification can be implemented in combination with the labeling process of the first embodiment and other modifications as post-processing. Whether or not to implement data cleansing according to this modification may be determined by a user's instruction after a graph of the time transition of fluorescence intensity after labeling, such as that shown in Figure 13, is displayed on a display.

[0089] (Modification 6 of the First Embodiment: Analysis of Information Indicating Whether or Not a Cell Candidate Region is a Differentiated Region) As the state of the cell candidate region, it is also preferable to analyze information indicating whether or not it is a differentiated region in step S106. That is, in this modification, the feature amounts calculated in steps S105 and S104 are used to acquire a true / false value or the like indicating whether or not the cell candidate region is a differentiated region in step S106.

[0090] When culturing cells with differentiation potential, such as embryonic stem cells (ES cells), induced pluripotent stem cells (iPS cells), and somatic stem cells, it is known that inappropriate culture conditions can result in the generation of cells in an advanced state of differentiation. The "differentiation region" refers to this region in an advanced state of differentiation. To generate a learning model for analyzing information indicating whether a region is differentiated, a time-series set of fluorescent images is acquired by fluorescently staining undifferentiated and differentiated markers. For undifferentiated markers, fluorescent staining reagents for rBC2LCN and various pluripotent stem cell markers are used for ES cells and iPS cells, and for somatic stem cells, fluorescent staining reagents for the corresponding somatic stem cell markers are used. As differentiation markers, three germ layer markers can be selected in the case of ES cells and iPS cells, and markers corresponding to differentiated cells can be selected in the case of somatic stem cells, with reference to a differentiation lineage diagram. The other steps are the same as those in the first embodiment and will not be described here. This modification makes it possible to analyze information relating to the state of the cell, such as a true / false value indicating whether or not a cell candidate region is a differentiated region.

[0091] (Seventh Modification of the First Embodiment: Unsupervised Clustering Using Time-Series Data as Input) In the first embodiment, a method for performing unsupervised clustering based on the fluorescence intensity feature data of each time-series cell candidate region group calculated as shown in FIG. 11 has been described. However, this is not limiting, and an unsupervised clustering method that takes time-series data as input and divides it into any number of clusters may also be used. For example, a method such as TimeSeriesKMeans can be used. When using a method that takes time-series data as input, the time-series data group of fluorescence intensity for each time-series cell candidate region group, as shown in FIG. 8, becomes the target data for unsupervised clustering.

[0092] This modification is effective when it is difficult to estimate a feature that is particularly useful for unsupervised clustering, such as the fluorescent brightness at a specific timing or the statistical value of the amount of change in the fluorescent brightness, as shown in FIG.

[0093] (Reference example: Accuracy evaluation of trained models) In the first embodiment, an example of training a trained model has been described. However, the accuracy of the trained model may be evaluated by preparing a data group for accuracy evaluation that is different from the data used for training. Here, the data group for accuracy evaluation may be a data group extracted by dividing a portion of the data group used for training. For example, data is randomly extracted so that the ratio of the number of data for training to the number of data for accuracy evaluation is 8:2. When the data group to be used for training and accuracy evaluation of the trained model is a data group acquired for multiple different culture vessels, the data group may be divided into a training data group and an accuracy evaluation data group on a culture vessel basis, rather than being randomly extracted from the entire data group. In this reference example, an example of accuracy evaluation of a trained model will be described in which a live cell determination model that determines whether a cell candidate region is a live cell or not is generated as the trained model. The live cell determination model can be trained by an anomaly detection-based machine learning model that classifies into two classes, namely, the "live cell" state and the "other" state, as described in the first embodiment. The accuracy evaluation can be performed by comparing the correct labels assigned to the accuracy evaluation data group by the processing based on steps S601 to S607 with information on the state of the cells in the cell candidate region acquired based on steps S101 to S106. More specifically, it is sufficient to calculate indices such as sensitivity, specificity, and F1-score, which are evaluation indices commonly used in anomaly detection.

[0094] FIG. 30 is a table showing an example of the results of evaluating the accuracy of live cell determination models. In the first embodiment, an example was described in which a feature vector composed of the diameter, average brightness, minimum brightness, brightness deviation, average movement amount, and diameter fluctuation amount of each cell candidate region in a cell image, as shown in FIG. 15, was used as input data to the trained model. In the example of FIG. 30, the accuracy of three live cell determination models No. 1, No. 2, and No. 3, which vary the types and combinations of feature values ​​that make up the input data, are compared. The circles in FIG. 30 indicate which feature values ​​each live cell determination model has as feature values ​​that make up the input feature vector. The accuracy rate (viable cells) and accuracy rate (non-viable cells) are synonymous with specificity and recall, respectively, which are commonly used evaluation metrics for anomaly detection models.Furthermore, the F1-score is the harmonic mean of precision and recall, and is commonly used as an evaluation metric for anomaly detection models, just like sensitivity and specificity.

[0095] Figure 31 is a diagram showing the results of evaluating the accuracy rate (viable cells) for each culture time. Graphs 3101, 3102, and 3103 in graph 3100 correspond to model Nos. 1, 2, and 3 in table 3000 in Figure 30, respectively. Figure 32 is a diagram showing the results of evaluating the accuracy rate (non-viable cells) for each culture time. Graphs 3201, 3202, and 3203 in graph 3200 correspond to model Nos. 1, 2, and 3 in table 3000 in Figure 30, respectively. The cell image analysis method according to the first embodiment may further include a display control step of displaying the results of the accuracy evaluation according to this reference example. In this case, it is preferable to display information about the data group used in training the trained model and evaluating the accuracy, as well as numerical values ​​or graphs representing the accuracy evaluation results, such as those shown in Figures 30, 31, and 32. The information about the data group includes, for example, information indicating culture conditions such as the number of days the cell sample was cultured and the seeding density, information indicating imaging conditions such as the imaging field of view and exposure time, and information indicating the number of data, such as the total number of data and the number of data for each correct label.

[0096] (Second Reference Example: Accuracy Evaluation of Trained Models) The aforementioned reference example explained an example in which the accuracy evaluation results of three live cell determination models trained by changing the types and combinations of features that make up the input data were compared with an anomaly detection-based machine learning model. In this reference example, as another suitable example, we will explain a comparison of the accuracy evaluation results of four live cell determination models trained by changing the types and combinations of features constituting the input data against a machine learning model using a decision tree based on the CART algorithm. As the evaluation indexes for the accuracy evaluation, the accuracy rate (live cells), accuracy rate (non-live cells), and F1-Score were calculated, as in Figure 30.

[0097] Figure 33 is a table showing an example of the results of evaluating the accuracy of live cell determination models. The accuracy of four live cell determination models, No. 1, No. 2, No. 3, and No. 4, which vary the types and combinations of features that make up the input data, is compared. As in Figure 30, the circles indicate which features each live cell determination model has as features that make up the input feature vector. Here, brightness features collectively refer to the average brightness value, minimum brightness value, and brightness deviation of the data shown in Figure 15. 33 shows the results of dividing the accuracy evaluation data group into two data groups, one for the culture time from 0 to 11 hours and the other for 12 hours and beyond, and calculating the accuracy evaluation index for each data group for each culture time period. In this way, accuracy evaluation values ​​divided by culture time period may be calculated and displayed. Figure 34 is a diagram showing the results of evaluating the accuracy rate (viable cells) for each culture time. Graphs 3401, 3402, 3403, and 3404 in graph 3400 correspond to Model Nos. 1, 2, 3, and 4, respectively, in table 3300 shown in Figure 33. Figure 35 is a diagram showing the results of evaluating the accuracy rate (non-viable cells) for each culture time. Graphs 3500 shown in Figure 35, 3501, 3502, 3503, and 3504 in graph 3500 shown in Figure 35, correspond to Model Nos. 1, 2, 3, and 4, respectively, in table 3300 shown in Figure 33. As described above, this reference example allows users to understand the performance of the trained model.

[0098] 30 to 35 used to explain the reference example show accuracy evaluation results for different models at time periods or predetermined times. In the analysis step according to the first embodiment, the trained model may be switched according to time based on these accuracy evaluation results. For example, the trained model with the highest F1-Score is selected for each time period. In the example of FIG. 33, model No. 2 is applied to the data group with an incubation time of 0 to 11 hours, and model No. 4 is applied to the data group from 12 hours onward. Alternatively, the user may refer to the accuracy evaluation results shown in FIGS. 30 to 35 and determine in advance which model to apply to which time period.

[0099] Second Embodiment (overview) In the first embodiment, a method was described in which a feature vector composed of brightness features and morphological features calculated from a group of time-series phase-contrast images observed in a time series, or features related to their changes over time, is input to a trained model to obtain information about the state of a cell. In this embodiment, we will explain a method for obtaining information about the state of cells by inputting time-series data, which is time-dependent information such as brightness features and morphological features calculated from a group of time-series phase-contrast images observed in a time series, into a trained model.

[0100] (Processing Procedure) FIG. 20 shows a flowchart of the cell information acquisition method according to the second embodiment. Steps S2001 to S2004 are the same as steps S101 to S104 in the first embodiment, and therefore will not be described further. Steps S2001 to S2004 acquire time-series data of brightness features and morphological features in each time-series cell candidate region group, and this information is output to an execution unit (e.g., analysis unit 2804) in step S2005.

[0101] (Step S2005: Analysis process) Step S2005 is a process of acquiring information about the state of the cells in the cell candidate region using the time-series data of the brightness feature amount and morphological feature amount calculated in steps S2001 to S2004.

[0102] In this step, a trained model is used that is pre-trained to input time-series data of features such as brightness features and morphological features calculated in S2004 and output information about the state of cells in the cell candidate region. The information about the state of cells in the cell candidate region is, for example, a true / false value indicating whether the cell candidate region is in a live cell, dead cell, or non-cellular state, as in the first embodiment.

[0103] In the machine learning model of this embodiment, time-series feature data obtained by extracting time-series data of features for a certain time period in the past from an arbitrary timing is used as input data. For example, time-series feature data for a time period of the past 12 hours is used as input. As a specific example, results of extracting time-series feature data that serves as input to the machine learning model at the timing of 12 hours of culture are shown in FIGS. 21A to 21C. Table 2100 shown in FIG. 21A is a feature data group acquired in step S2004. Table 2101 shown in FIG. 21B is a result of extracting time-series feature data at the timing of 12 hours of culture from the feature data group of Table 2100 in FIG. 21A. Feature data for the corresponding time period is extracted for each time-series cell candidate region group having the same tracking number, such as time-series feature data DT001 or DT002. In this embodiment, data such as DT001 and DT002 shown in Figure 21B are input into the trained model, and for each, true / false value results RT001 (corresponding to DT001) and RT002 (corresponding to DT002) are obtained, indicating whether the cell candidate region shown in 2102 in Figure 21C is in a live cell, dead cell, or non-cell state.

[0104] Below, we will explain the specific method for generating a trained model. As in the first embodiment, the trained model is a machine learning model trained using as training data the results of a labeling process based on information about fluorescence brightness, including at least information about changes in fluorescence brightness over time.

[0105] (Processing procedure for generating trained models) FIG. 22 is a flow diagram showing the flow of generating a trained model according to the analysis step of the second embodiment.

[0106] Steps S2201 to S2204 are the same as steps S601 to S604 in the first embodiment, and therefore will not be described further. Steps S2201 to S2204 obtain time-series data of brightness features and morphological features in each time-series cell candidate region group, and output the data to the execution unit in step S2205.

[0107] Steps S2205 and S2206 are the same processes as steps S606 and S607 in the first embodiment, and therefore will not be described further. Steps S2205 and S2206 output data, to the execution unit of step S2207, in which correct labels relating to the state of cells in the cell candidate regions are assigned to the time-series data of brightness features and morphological features in each time-series cell candidate region group.

[0108] (Step S2207: Generate a trained model) Step S2207 is a trained model generation step for generating a trained model based on the feature data group of each cell candidate region to which information regarding the state of the cell obtained in step S2206 has been added.

[0109] In this step, time-series data of feature quantities in the cell candidate regions of the phase-contrast images included in each time-series cell candidate region group is input, and using the labels assigned in step S2206 as training, a machine learning model is trained to output a true / false value indicating whether the cell is live, dead, or non-cellular, thereby generating a trained model. The machine learning model may be any machine learning model that inputs time-series data of feature quantities and classifies the data into any number of classes. For example, a classification model using a neural network such as an RNN can be used. Alternatively, a neural network model such as an LSTM can be used. It is also possible to treat time-series data of multiple types of feature quantities as two-dimensional array data and train a neural network for image classification, such as ResNet. The method for generating a trained model related to the analysis step of the second embodiment has been described above.

[0110] In step S2006, the time-series feature amount data as shown in FIG. 22 is input to the trained model generated in steps S2201 to S2207, and information about the state of the cells in the cell candidate region is acquired.

[0111] As described above, this embodiment makes it possible to present to the user the judgment results regarding the state of cells using a trained model based on more reliable training data.

[0112] <Third embodiment> (overview) In the first embodiment, a method was described in which a feature vector composed of brightness features and morphological features calculated from a group of time-series phase-contrast images observed in a time series, or features related to their changes over time, is input to a trained model to obtain information about the state of a cell. In this embodiment, a method is described in which a partial cell image obtained by cutting out a cell region corresponding to a cell candidate region from a phase contrast image is used as input to a trained model to obtain information about the state of the cell.

[0113] (Processing Procedure) FIG. 23 shows a flowchart of the cell information acquisition method according to the third embodiment. Steps S2301 to S2303 are the same as steps S101 to S103 in the first embodiment, and therefore will not be described further. Steps S2301 to S2303 assign region tracking numbers, which indicate which cell candidate regions are temporally continuous and identical, to the mask image and contour coordinate information corresponding to each cell region, and output this information to an execution unit (e.g., analysis unit 2804) in step S2304.

[0114] (Step S2304: Analysis process) Step S2304 is an analysis step for acquiring information about the state of the cells in the cell candidate region based on the region tracking number and the cell candidate region acquired in step S2303.

[0115] In this step, a pre-trained model is used that inputs a partial cell image of the cell candidate region based on the region tracking number and the cell candidate region in S2304 and outputs information about the state of the cells in the cell candidate region. Here, the partial cell image is an image obtained by cutting out a rectangular region of a predetermined size from the cell image, with the center of gravity of the cell candidate region at its center. The predetermined size may be any size that includes the target cell candidate region; for example, a rectangular region of 64 x 64 pixels is cut out. Furthermore, the information about the state of the cells in the cell candidate region is, for example, a true / false value indicating whether the cell candidate region is a live cell, a dead cell, or a non-cell, as in the first embodiment.

[0116] In this embodiment, the machine learning model uses a set of partial cell images extracted from a certain time period in the past from an arbitrary timing as input data. For example, a set of partial cell images from the past 12 hours is used as input. As a specific example, FIGS. 24A and 24B show the results of extracting a set of partial cell images that serve as input to the machine learning model at a time point of 12 hours of culture time. Reference numeral 2401 in FIG. 24A denotes a set of partial cell images extracted from each cell image based on the region tracking number and cell candidate region acquired in step S2304. For example, at a time point of 12 hours of culture time, partial cell images such as 2401a and 2401b serve as input to the machine learning model. The partial cell images such as 2401a and 2401b are input to the trained model, and a true / false result is obtained for each set of partial cell images, indicating whether the cell candidate region, as shown in 2402a (corresponding to 2401a) and 2402b (corresponding to 2401b) in 2402 of FIG. 24B, is in a live cell, dead cell, or non-cell state.

[0117] Below, we will explain the specific method for generating a trained model. As in the first embodiment, the trained model is a machine learning model trained using as training data the results of a labeling process based on information about fluorescence brightness, including at least information about changes in fluorescence brightness over time.

[0118] (Processing procedure for generating trained models) FIG. 25 is a flow diagram showing the flow of generating a trained model according to the analysis step of the third embodiment.

[0119] Steps S2501 to S2503 are the same as steps S601 to S603 in the first embodiment, and therefore will not be described further. Steps S2501 to S2503 assign region tracking numbers, which indicate which cell candidate regions are temporally continuous cell candidate regions, to the mask image and contour coordinate information corresponding to each cell region, and this information is output to the execution unit in step S2504.

[0120] Steps S2504 and S2505 are the same processes as steps S606 and S607 in the first embodiment, and therefore description thereof will be omitted. Steps S2404 and S2405 output data to each cell candidate region, to which a correct answer label relating to the state of the cell in the cell candidate region is assigned, to the execution unit (e.g., analysis unit 2804) of step S2506.

[0121] (Step S2506: Generate a trained model) Step S2506 is a trained model generation step for generating a trained model based on each cell candidate region to which information regarding the state of the cell obtained in step S2505 has been added.

[0122] In this step, a group of partial cell images extracted from a certain time period in the past from an arbitrary timing is input, and using the labels assigned in step S2506 as training, a machine learning model is trained to output a true / false value indicating whether the cell is alive, dead, or non-cellular, generating a trained model. The machine learning model can be a neural network that inputs time-series images, such as ConvLSTM. It is also possible to treat the group of partial cell images, which are time-series image data, as 3D volume data and train a neural network model, such as 3D-ResNet, that classifies 3D volume data. The method for generating a trained model related to the analysis step of the third embodiment has been described above.

[0123] In step S2304, a group of partial cell images such as those shown in FIG. 24 is input to the trained model generated in steps S2501 to S2506, and information about the state of cells in the cell candidate region is obtained.

[0124] As described above, this embodiment makes it possible to present to the user the judgment results regarding the state of cells using a trained model based on more reliable training data.

[0125] <Cell information acquisition program / cell information acquisition system> The present disclosure provides a program for causing a computer to execute the cell image analysis method according to the present disclosure described above. The present disclosure also provides a medium storing the program in a computer-readable format.

[0126] FIG. 26 is a block diagram showing an example of the hardware configuration of an information processing device 2600 capable of executing the program according to the present invention.

[0127] The information processing device 2600 has computer functions. For example, the information processing device 2600 may be integrated with a desktop personal computer (PC), a laptop PC, a tablet PC, a smartphone, or the like.

[0128] The information processing device 2600 includes a central processing unit (CPU) 2601, a random access memory (RAM) 2602, a read only memory (ROM) 2603, and a hard disk drive (HDD) 2604 to function as a computer that performs calculations and storage. The information processing device 2600 also includes a communication interface (I / F) 2605, a display device 2606, and an input device 2607. The CPU 2601, RAM 2602, ROM 2603, HDD 2604, communication I / F 2605, display device 2606, and input device 2607 are interconnected via a bus 2610. The display device 2606 and input device 2607 may be connected to the bus 2610 via a driver (not shown) for driving these devices.

[0129] 26, the components constituting the information processing device 2600 are illustrated as an integrated device, but some of these functions may be configured by external devices. For example, the display device 2606 and the input device 2607 may be external devices separate from the components constituting the functions of the computer, including the CPU 2601, etc.

[0130] The CPU 2601 performs predetermined operations in accordance with programs stored in the RAM 2602, HDD 2604, etc., and also has the function of controlling each unit of the information processing device 2600. The RAM 2602 is made up of a volatile storage medium and provides a temporary memory area necessary for the operation of the CPU 2601. The ROM 2603 is made up of a non-volatile storage medium and stores necessary information such as programs used for the operation of the information processing device 2600. The HDD 2604 is a storage device made up of a non-volatile storage medium.

[0131] The communication I / F 2605 is a communication interface based on standards such as Wi-Fi (registered trademark) and 4G, and is a module for communicating with other devices. The display device 2606 is a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or the like, and is used to display moving images, still images, characters, and the like. The input device 2607 is a button, a touch panel, a keyboard, a pointing device, or the like, and is used by the user to operate the information processing device 2600. The display device 2606 and the input device 2607 may be integrally formed as a touch panel.

[0132] Note that the hardware configuration shown in FIG. 26 is an example, and other devices may be added, or some devices may not be provided. Also, some devices may be replaced with other devices having similar functions. Furthermore, some functions may be provided by other devices via a network, or the functions constituting this embodiment may be distributed and realized among multiple devices. For example, the HDD 2604 may be replaced with an SSD (Solid State Drive) using semiconductor elements such as flash memory, or may be replaced with cloud storage.

[0133] <Cell image analysis device> 28 , the present disclosure provides a cell image analysis device including an image acquisition unit 2801 that acquires a time-series cell image group by collecting multiple cell images of cells captured at different consecutive times using either bright-field observation or phase-contrast observation, and associating the multiple cell images with the multiple times, a region extraction unit 2802 that extracts cell candidate regions from the cell images included in the time-series cell image group, a region tracking unit 2803 that determines whether the cell candidate regions in multiple cell images included in the time-series cell image group, each associated with a different time, correspond to the same target, and collects the cell candidate regions determined to correspond to the same target in association with the time, thereby acquiring them as a time-series cell candidate region group, and an analysis unit 2804 that analyzes information about the state of the cells based on information about the time-series cell candidate region group. The analysis unit uses a trained model, and the trained model is a machine learning model trained based on information about the time-series cell candidate region group and information about the state of the cells acquired from training cells, using information about the time-series cell candidate region group as input and information about the state of the cells as output. The image acquisition unit 2801, the region extraction unit 2802, the region tracking unit 2803, and the analysis unit 2804 respectively execute the image acquisition step, the region extraction step, the region tracking step, and the analysis step described above.

[0134] <Cell image analysis system> The present disclosure also provides a cell image analysis system including an image acquisition device 2901 and an information processing device 2902, as shown in FIG. The image acquisition device 2901 acquires cell images by photographing cells at multiple consecutive different timings using either bright-field observation or phase-contrast observation, and the information processing device 2902 includes an image acquisition unit 2903 that acquires the cell images from the image acquisition device 2901 and acquires a time-series cell image group obtained by collecting the cell images in association with the timings; a region extraction unit 2904 that extracts cell candidate regions from the cell images included in the time-series cell image group; a region tracking unit 2905 that determines whether the cell candidate regions of multiple cell images included in the time-series cell image group and associated with different timings correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timings, and acquires them as a time-series cell candidate region group; and an analysis unit 2906 that analyzes information about the state of the cells based on information about the time-series cell candidate region group, and the analysis unit uses a trained model for learning cells that receives information about the time-series cell candidate region group as input and outputs information about the state of the cells.

[0135] The image acquisition device 2901 can capture images of cells at multiple consecutive different timings using either bright-field observation or phase-contrast observation. Preferably, the image acquisition device 2901 can include a lens, a detector, a camera, an excitation light, a storage device, etc. Preferred examples of the image acquisition device 2901 include a phase-contrast microscope, a fluorescence microscope, and a confocal microscope. The information processing device is as described above. The image acquisition unit 2801, the region extraction unit 2802, the region tracking unit 2803, and the analysis unit 2804 execute the image acquisition step, the region extraction step, the region tracking step, and the analysis step, respectively, as described above.

[0136] Embodiments of the present disclosure include the following methods and configurations. (Method 1) an image acquisition step of acquiring a time-series cell image group in which a plurality of cell images are acquired by photographing the cells at different consecutive times using either bright-field observation or phase-contrast observation and are collected in association with the times; a region extraction step of extracting a cell candidate region from the cell image included in the time-series cell image group; a region tracking step of determining whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, and collecting the cell candidate regions determined to correspond to the same object in association with the timing, thereby acquiring the cell candidate regions as a time-series cell candidate region group; an analysis step of analyzing information about a state of a cell based on information about the time-series cell candidate region group, the analyzing step is characterized by using a trained model; The trained model is generated based on information about the time-series cell candidate region group obtained from the training cells and information about the state of the cells. A cell image analysis method characterized by being a machine learning model trained using information about the time-series cell candidate region group as input and information about the state of the cells as output. (Method 2) The cell image analysis method described in Method 1, characterized in that in the trained model, information regarding the state of the cell is obtained based on fluorescence brightness feature data that includes at least information regarding changes in fluorescence brightness over time. (Method 3) The trained model includes, for the training cells, a fluorescence image acquisition step of acquiring a time-series fluorescence image group in which fluorescence images corresponding to the cell images included in the time-series cell image group are collected in association with the timing; a fluorescence brightness feature calculation step of acquiring fluorescence brightness feature data of the time-series cell candidate region group based on the time-series fluorescence image group; a labeling process step of assigning a label relating to a state of a cell to the cell candidate region based on the fluorescence brightness feature amount data; and a trained model generation step of generating the trained model by training the trained model using information about the cell state based on the label related to the cell state as an output and information about the time-series cell candidate region group as an input; The invention is characterized in that it is obtained by carrying out moreover, 3. The cell image analysis method according to Method 1 or 2, wherein the fluorescence brightness feature data includes at least information regarding changes in fluorescence brightness over time. (Method 4) A cell image analysis method described in any one of methods 1 to 3, characterized in that the information regarding the state of the cell includes at least information regarding a true / false value indicating whether the cell candidate region is a living cell region or a scalar value indicating the possibility. (Method 5) A cell image analysis method described in any one of methods 1 to 3, characterized in that the information regarding the state of the cell includes at least information regarding a true / false value indicating whether the cell candidate region is a differentiated region or a scalar value indicating the possibility. (Method 6) the information about the time-series cell candidate region group is feature amount data included in the time-series cell image group, 6. The cell image analysis method according to any one of methods 1 to 5, wherein the feature data includes one or more selected from the group consisting of brightness features in the cell candidate region of the cell image, morphological features of the cell candidate region, information on changes in the brightness features over time, and information on changes in the morphological features over time, and the feature data is a scalar value or a vector. (Method 7) 6. The cell image analysis method according to claim 1, wherein the information about the time-series cell candidate region group is time-series feature data collected by associating feature data of the cell candidate region included in the time-series cell image group with the timing, and the feature data includes one or more selected from the group consisting of brightness feature data of the cell candidate region of the cell image and morphological feature data of the cell candidate region. (Method 8) 6. The cell image analysis method according to claim 1, wherein the information about the time-series cell candidate region group is time-series image data collected by associating partial cell images corresponding to the cell candidate regions of the cell images included in the time-series cell image group with the timing. (Configuration 1) an image acquisition unit that acquires a time-series cell image group in which a plurality of cell images are acquired by capturing images of cells at different consecutive times using either bright-field observation or phase-contrast observation, and the acquired images are associated with the times; a region extraction unit that extracts a cell candidate region from the cell image included in the time-series cell image group; a region tracking unit that determines whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timing, and acquires them as a time-series cell candidate region group; an analysis unit that analyzes information about a state of a cell based on information about the time-series cell candidate region group; The analysis unit uses a trained model, The trained model is generated based on information about the time-series cell candidate region group obtained from the training cells and information about the state of the cells. A cell image analysis device characterized by being a machine learning model trained using information about the time-series cell candidate region group as input and information about the state of the cells as output. (Configuration 2) A program for causing a computer to execute the method according to any one of methods 1 to 8. (Configuration 3) A medium storing the program according to configuration 2 in a computer-readable format. (Configuration 4) The image acquisition device and the information processing device are included. the image acquisition device acquires cell images by photographing cells at multiple consecutive different timings using either bright-field observation or phase-contrast observation; The information processing device includes: an image acquisition unit that acquires the cell images from the image acquisition device and acquires a time-series cell image group that is collected by associating the cell images with the timing; a region extraction unit that extracts a cell candidate region from the cell image included in the time-series cell image group; a region tracking unit that determines whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timing, and acquires them as a time-series cell candidate region group; an analysis unit that analyzes information about a state of a cell based on information about the time-series cell candidate region group; The analysis unit uses a trained model for a training cell that receives information about the time-series cell candidate region group as input and outputs information about the state of the cell.

Claims

1. an image acquisition step of acquiring a time-series cell image group in which a plurality of cell images are acquired by photographing the cells at different consecutive times using either bright-field observation or phase-contrast observation and are collected in association with the times; a region extraction step of extracting a cell candidate region from the cell image included in the time-series cell image group; a region tracking step of determining whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, and collecting the cell candidate regions determined to correspond to the same object in association with the timing, thereby acquiring the cell candidate regions as a time-series cell candidate region group; an analysis step of analyzing information about a state of a cell based on information about the time-series cell candidate region group, the analyzing step is characterized by using a trained model; The trained model is generated based on information about the time-series cell candidate region group obtained from the training cells and information about the state of the cells. A cell image analysis method characterized by being a machine learning model trained using information about the time-series cell candidate region group as input and information about the state of the cells as output.

2. The cell image analysis method according to claim 1, wherein in the trained model, information regarding the state of the cell is obtained based on fluorescence brightness feature data including at least information regarding changes in fluorescence brightness over time.

3. The trained model includes, for the training cells, a fluorescence image acquisition step of acquiring a time-series fluorescence image group in which fluorescence images corresponding to the cell images included in the time-series cell image group are collected in association with the timing; a fluorescence brightness feature calculation step of acquiring fluorescence brightness feature data of the time-series cell candidate region group based on the time-series fluorescence image group; a labeling process step of assigning a label relating to a state of a cell to the cell candidate region based on the fluorescence brightness feature amount data; and a trained model generation step of generating the trained model by training the trained model using information about the cell state based on the label related to the cell state as an output and information about the time-series cell candidate region group as an input; The invention is characterized in that it is obtained by carrying out moreover, The cell image analysis method according to claim 2 , wherein the fluorescence luminance feature amount data includes at least information relating to a change in fluorescence luminance over time.

4. The cell image analysis method according to claim 1, characterized in that the information regarding the state of the cell includes at least information regarding a true / false value indicating whether or not the cell candidate region is a live cell region, or a scalar value indicating a possibility.

5. The cell image analysis method according to claim 1, characterized in that the information regarding the state of the cell includes at least information regarding a true / false value indicating whether the cell candidate region is a differentiated region or a scalar value indicating a possibility.

6. the information about the time-series cell candidate region group is feature amount data included in the time-series cell image group, 6. The cell image analysis method according to claim 1, wherein the feature data includes one or more selected from the group consisting of brightness feature values ​​in the cell candidate region of the cell image, morphological feature values ​​in the cell candidate region, information on temporal changes in the brightness feature values, and information on temporal changes in the morphological feature values, and the feature data is a scalar value or a vector.

7. 6. The cell image analysis method according to claim 1, wherein the information about the time-series cell candidate region group is time-series feature data collected by associating feature data of the cell candidate region included in the time-series cell image group with the timing, and the feature data includes one or more selected from the group consisting of brightness feature data of the cell candidate region of the cell image and morphological feature data of the cell candidate region.

8. The cell image analysis method according to any one of claims 1 to 5, characterized in that the information relating to the time-series cell candidate region group is time-series image data collected by associating partial cell images corresponding to the cell candidate regions of the cell images included in the time-series cell image group with the timing.

9. an image acquisition unit that acquires a time-series cell image group in which a plurality of cell images are acquired by capturing images of cells at different consecutive times using either bright-field observation or phase-contrast observation, and the acquired images are associated with the times; a region extraction unit that extracts a cell candidate region from the cell image included in the time-series cell image group; a region tracking unit that determines whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timing, and acquires them as a time-series cell candidate region group; an analysis unit that analyzes information about a state of a cell based on information about the time-series cell candidate region group; The analysis unit uses a trained model, The trained model is generated based on information about the time-series cell candidate region group obtained from the training cells and information about the state of the cells. A cell image analysis device characterized by being a machine learning model trained using information about the time-series cell candidate region group as input and information about the state of the cells as output.

10. A program for causing a computer to execute the method according to any one of claims 1 to 5.

11. A medium storing the program according to claim 10 in a computer-readable format.

12. The image acquisition device and the information processing device are included. the image acquisition device acquires cell images by photographing cells at multiple consecutive different timings using either bright-field observation or phase-contrast observation; The information processing device includes: an image acquisition unit that acquires the cell images from the image acquisition device and acquires a time-series cell image group that is collected by associating the cell images with the timing; a region extraction unit that extracts a cell candidate region from the cell image included in the time-series cell image group; a region tracking unit that determines whether the cell candidate regions of a plurality of the cell images included in the time-series cell image group, each associated with a different timing, correspond to the same object, collects the cell candidate regions determined to correspond to the same object in association with the timing, and acquires them as a time-series cell candidate region group; an analysis unit that analyzes information about a state of a cell based on information about the time-series cell candidate region group; The analysis unit uses a trained model for a training cell that receives information about the time-series cell candidate region group as input and outputs information about the state of the cell.

Images