Determination device, determination method, and determination program

Abstract

To provide a determination device, a determination method, and a determination program for determining whether or not an anomaly exists in culture of cells photographed in a photographed image.SOLUTION: A determination device 10 includes: an acquisition unit 110 that acquires a photographed image obtained by photographing a field of view including cells; a calculation unit 115 that calculates a non-normality degree of the photographed image by using, as a reference, one or a plurality of reference images obtained by photographing normally cultured cells; and a determination unit 130 that determines whether or not an anomaly exists in culture of the cells photographed in the photographed image based on the non-normality degree.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a determination device, a determination method, and a determination program. [Background technology]

[0002] Patent Document 1 describes a cell evaluation device comprising: an image reading unit that reads in a plurality of images of a plurality of cells cultured in a culture vessel in a time series; a feature amount calculation unit that obtains, from the images, a plurality of different feature amounts that indicate a plurality of different morphological features of the cells for each of the cells included in the images; a feature amount determination unit that determines, for each of the feature amounts corresponding to each of the plurality of images, whether it is appropriate to evaluate the characteristics to be evaluated for the cells using the feature amount; a continuity determination unit that determines, for each of the feature amounts, whether the determination results by the feature amount determination unit for the feature amount are continuous in the time series; and a computational model construction unit that combines the feature amount obtained by the feature amount calculation unit and the determination results by the continuity determination unit to construct a computational model for evaluating the characteristics of the cells (claim 1). Patent Document 2 describes an image processing device that includes a CNN processing unit that uses a convolutional neural network to extract image features of the bright field image and the fluorescent image and output identification information related to the cells (claim 1). Furthermore, Patent Document 2 states that "the identification information relating to the cell includes information relating to the type, morphology, or distribution of the cell, or the type, morphology, or distribution of biological material contained within the cell" (Claim 2). [Prior art document] [Patent documents] [Patent Document 1] JP 2011-229410 A [Patent Document 2] JP 2018-180635 A Summary of the Invention

[0003] In a first aspect of the present invention, a determination device is provided. The determination device may include an acquisition unit that acquires a captured image of a visual field including cells. The determination device may include a calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The determination device may include a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality.

[0004] The determination device may include an analysis processing unit that generates an analysis image that displays a distribution of influences on the determination on the captured image.

[0005] The calculation unit may include a reconstructed image generation unit that generates a reconstructed image by reconstructing the captured image so as to resemble one or more reference images. The calculation unit may include an anomaly calculation unit that calculates an anomaly degree based on a difference between the captured image and the reconstructed image.

[0006] The reconstructed image generation unit may extract features of each of the one or more reference images, and input the captured image into a reconstruction model trained to restore each reference image from the extracted features, thereby generating a reconstructed image.

[0007] The determination device may include a learning processing unit that performs learning processing of the reconstruction model using one or more reference images obtained by capturing a visual field including cells that have been determined to have been normally cultured.

[0008] The learning processing unit may add a captured image that has not been determined by the determination unit to have had an abnormality in the cell culture as a new reference image to a set of one or more reference images, and perform learning processing of the reconstruction model.

[0009] The determination device may include a plurality of calculation units. A plurality of reconstructed image generation units included in the plurality of calculation units may input the captured image to a plurality of different reconstructed models, respectively, to generate a plurality of reconstructed images. A plurality of abnormality degree calculation units included in the plurality of calculation units may calculate an abnormality degree for each of the plurality of reconstructed images. The determination unit may output a determination result that integrates the abnormality degrees of each of the plurality of reconstructed images.

[0010] The calculation unit may include a feature amount calculation unit that calculates a feature amount of the captured image. The calculation unit may include an anomaly degree calculation unit that calculates an anomaly degree based on the feature amount of the captured image and each of the reference feature amounts of one or more reference images.

[0011] The acquisition unit may acquire images of the cell culture process captured in time series. The calculation unit may calculate the degree of abnormality based on one or more reference images of the normal cell culture process captured in time series.

[0012] The determination device may include a pre-processing unit that performs pre-processing on the captured image. The calculation unit may calculate the degree of abnormality using the captured image that has been subjected to the pre-processing.

[0013] The pre-processing may include at least one of a three-dimensional reconstruction process or an edge enhancement process of the image of the cell or cell cluster.

[0014] The acquisition unit may acquire a photographed image for each of the multiple visual fields. The determination device may include multiple calculation units. The calculation units may calculate a degree of abnormality of each of the photographed images for the multiple visual fields. The determination unit may output a determination result that integrates the degrees of abnormality of each of the photographed images for the multiple visual fields.

[0015] The determination unit may output a determination result that integrates the degree of abnormality for each culture plate or well.

[0016] The determination unit may output a determination result that integrates the degrees of abnormality for a plurality of culture plates or wells.

[0017] The acquisition unit may further acquire the culture conditions of the cells. The calculation unit may calculate the degree of abnormality based on one or more reference images of cells normally cultured under culture conditions corresponding to the acquired culture conditions.

[0018] According to a second aspect of the present invention, there is provided a determination method. The determination method may include a determination device acquiring a captured image of a field of view including cells. The determination method may include a determination device calculating a degree of abnormality of the captured image based on one or more reference images capturing normally cultured cells. The determination method may include a determination device determining whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality.

[0019] According to a third aspect of the present invention, there is provided a judgment program executed by a computer. The judgment program may cause the computer to function as an acquisition unit that acquires a captured image of a visual field including cells. The judgment program may cause the computer to function as a calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The judgment program may cause the computer to function as a judgment unit that judges whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality.

[0020] The above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0021] [Figure 1] 1 shows a configuration of a determination device 10 according to the present embodiment. [Diagram 2] 2 shows a determination flow of the determination device 10 according to the present embodiment. [Diagram 3] 2 shows a learning flow of the determination device 10 according to the present embodiment. [Figure 4] 13 shows the configuration of a calculation unit 415 according to a first modified example of this embodiment. [Diagram 5] 13 shows an operational flow of a calculation unit 415 according to a first modified example of this embodiment. [Figure 6] 6 shows the configuration of a determination device 600 according to a second modified example of this embodiment. [Figure 7] 7 shows the configuration of a determination device 700 according to a third modified example of this embodiment. [Figure 8] 13 shows the configuration of a determination device 800 according to a fourth modified example of the present embodiment. [Figure 9] 13 shows a determination flow of a determination device 800 according to a fourth modified example of the present embodiment. [Figure 10] 13 shows the configuration of a determination device 1000 according to a fifth modified example of the present embodiment. [Figure 11] 13 shows a determination flow of a determination device 1000 according to a fifth modified example of the present embodiment. [Figure 12] 22 illustrates an example computer 2200 in which aspects of the present invention may be embodied, in whole or in part. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0023] 1 shows the configuration of a determination device 10 according to this embodiment. Many aspects of bioprocesses such as cell culture have not been fully elucidated. For this reason, even if the same protocol is used, there are variations in the culture process due to, for example, differences in the technique of the experimenter, differences in the equipment when the equipment is changed, the degree of deterioration of the raw cells, contamination, and other unexplained factors.

[0024] When using a model trained to extract features from images of normally cultured cells, if there is an abnormality in the culture of the target cells, there is a possibility that erroneous features will be extracted from the image generated due to the abnormality in the culture. The determination device 10 determines whether the cell was cultured normally or abnormally, thereby making it possible to indicate whether or not the cell can be used as a normally cultured cell, i.e., whether or not the image of the cell can be used to extract the features of the cell, for example.

[0025] The determination device 10 determines whether or not there is an abnormality in the culture by using a captured image of a measurement target 20 including cells. The determination device 10 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which multiple computers are connected. Such a computer system is also a computer in a broad sense. The determination device 10 may be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the determination device 10 may be a dedicated computer designed for determining the culture state of cells, or may be dedicated hardware realized by a dedicated circuit. When a computer is used, the determination device 10 is realized by executing a determination program by the computer. The determination device 10 includes an imaging unit 100, a captured image storage unit 105, an acquisition unit 110, a calculation unit 115, a determination unit 130, an analysis processing unit 132, a result output unit 135, and a model generation unit 137.

[0026] The imaging unit 100 captures an image of a visual field including cells. The captured image storage unit 105 is connected to the imaging unit 100 and stores the captured image captured by the imaging unit 100. The acquisition unit 110 is connected to the captured image storage unit 105 and acquires the captured image of the visual field including cells from the captured image storage unit 105. Note that in other embodiments, the determination device 10 may not include the imaging unit 100 and the captured image storage unit 105, and may acquire the captured image from an external imaging device or storage device.

[0027] The calculation unit 115 is connected to the acquisition unit 110. The calculation unit 115 calculates the degree of abnormality of a captured image based on one or more reference images of normally cultured cells. Here, the "degree of abnormality" is an index value indicating the degree to which the captured image deviates from a standard (standard) defined using one or more reference images. In this embodiment, the calculation unit 115 has a reconstructed image generation unit 120 and an abnormality calculation unit 125.

[0028] The reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image acquired by the acquiring unit 110 so as to resemble one or more reference images. This allows the reconstructed image generating unit 120 to output a reconstructed image from the captured image in which image features caused by culture abnormalities, such as those not seen in one or more reference images capturing normally cultured cells, have been removed or attenuated.

[0029] The abnormality degree calculation unit 125 is connected to the reconstructed image generation unit 120. The abnormality degree calculation unit 125 calculates the abnormality degree of the captured image based on the difference between the captured image and the reconstructed image. Here, since the reconstructed image is obtained by removing image features caused by culture abnormalities, the greater the image features caused by culture abnormalities are contained in the captured image, the greater the difference between the captured image and the reconstructed image. Therefore, in such a case, the "abnormality degree" is a value indicating a greater deviation from the standard.

[0030] The determination unit 130 is connected to the calculation unit 115. The determination unit 130 determines, based on the degree of abnormality calculated by the calculation unit 115, whether or not there is an abnormality in the culture of the cells captured in the captured image.

[0031] The analysis processing unit 132 is connected to the determination unit 130. The analysis processing unit 132 generates an analysis image on the captured image, which displays the distribution of influence on the determination by the determination unit 130. The analysis image indicates which area in the captured image contributed to the determination that there was an abnormality in the cell culture. For example, the analysis processing unit 132 generates an analysis image on the captured image, which displays the difference from the reconstructed image. Note that in other embodiments, the determination device 10 does not need to include the analysis processing unit 132 and does not need to have a function of generating an analysis image.

[0032] The result output unit 135 is connected to the analysis processing unit 132. The result output unit 135 outputs the determination result by the determination unit 130 and the analysis image generated by the analysis processing unit 132. For example, the result output unit 135 performs a display process to display the determination result and the analysis image on a display device or terminal used by a user. The result output unit 135 may store the determination result and the analysis image in a storage device.

[0033] The model generating unit 137 is connected to the acquiring unit 110. The model generating unit 137 generates a reconstruction model using one or more reference images obtained by capturing a field of view including cells determined to have been normally cultured. The model generating unit 137 according to the present embodiment selects a reference image from the captured images captured by the acquiring unit 110 and uses the reference image for generating a reconstruction model. Note that in other embodiments, the determination device 10 does not include the model generating unit 137, and the result output unit 135 may generate a reconstruction image using a predetermined reconstruction model or an externally provided reconstruction model.

[0034] The model generating unit 137 has a reference image storage unit 145, a learning processing unit 150, and a model storage unit 155. The reference image storage unit 145 stores one or more reference images obtained by photographing normally cultured cells.

[0035] The learning processing unit 150 is connected to the acquisition unit 110, the judgment unit 130, and the reference image storage unit 145. The learning processing unit 150 performs learning processing of a reconstruction model used by the reconstruction image generation unit 120 to reconstruct a captured image, using one or more reference images stored in the reference image storage unit 145. In this embodiment, the learning processing unit 150 adds, as a new reference image, a captured image that is not judged by the judgment unit 130 to have an abnormality in the cell culture, among the captured images acquired by the acquisition unit 110, to a set of one or more reference images stored in the reference image storage unit 145. Then, the learning processing unit 150 performs learning processing of the reconstruction model, using the one or more reference images to which the new reference image has been added. Note that, in other embodiments, the learning processing unit 150 may not have a function of online learning that adds a new reference image to the reference image storage unit 145 and updates the reconstruction model.

[0036] The model storage unit 155 is connected to the learning processing unit 150. The model storage unit 155 stores the reconstructed model generated by the learning processing unit 150. Then, the model storage unit 155 supplies the reconstructed model to the reconstructed image generation unit 120 in the calculation unit 115.

[0037] FIG. 2 shows a judgment flow of the judgment device 10 according to this embodiment. In step 200 (S200), the imaging unit 100 captures an image of a field including cells. The imaging unit 100 may be a microscope having an imaging function, for example. In this embodiment, the imaging unit 100 captures a bright field image of a field including cells in the measurement target 20. The imaging unit 100 may capture a phase contrast image of a field including cells, a stained image of a field including stained cells, or an observation image using various other techniques. The imaging unit 100 stores the captured image in the captured image storage unit 105. In S210, the acquisition unit 110 acquires the captured image from the captured image storage unit 105.

[0038] In S220, the reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image so as to resemble one or more reference images. In this embodiment, the reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image using a reconstruction model generated by the model generating unit 137 and stored in the model storage unit 155. Here, the learning processing unit 150 in the model generating unit 137 extracts features of each reference image among one or more reference images, and generates a reconstruction model by learning so as to restore each reference image from the extracted features. The reconstructed image generating unit 120 inputs the captured image into such a reconstruction model to generate a reconstructed image. Note that the reconstructed image generating unit 120 may reconstruct the captured image before reconstruction to generate a reconstructed image with the same resolution, or may generate a reconstructed image with a different resolution. In this way, the reconstructed image generating unit 120 reconstructs the captured image so as to resemble one or more reference images, thereby generating an image that serves as a reference (standard) in accordance with one or more reference images in the situation captured in the captured image.

[0039] In S230, the degree of abnormality calculation unit 125 calculates the degree of abnormality of the captured image based on the difference between the captured image and the reconstructed image. In this embodiment, the degree of abnormality calculation unit 125 uses the sum of squared errors obtained by summing up the squared errors of each pixel of the captured image and the reconstructed image as an index representing the degree of abnormality. That is, the degree of abnormality a(t) of a captured image x(t) captured at a certain time t is calculated by multiplying the reconstructed image by x rec If (t), it is expressed by the following equation (1): Here, the subscript i is a variable representing each pixel in the image.

number

[0040] The degree of abnormality a(t) expressed by formula (1) is an index value that increases as the difference between the captured image and the reconstructed image increases. The degree of abnormality a(t) calculated in this manner is an index value that indicates the degree of deviation of the captured image based on a reconstructed image conforming to one or more reference images in the situation in which the captured image was captured. Alternatively, the degree of abnormality may be an index value that decreases as the difference between the captured image and the reconstructed image increases. Note that when the resolutions of the captured image and the reconstructed image are different, the degree of abnormality calculation unit 125 may calculate the degree of abnormality a(t) by formula (1) using the difference between pixels at corresponding positions.

[0041] The abnormality degree calculation unit 125 may calculate the abnormality degree using a calculation method different from the above. rec Any statistical quantity that changes depending on the difference between the captured image and the reconstructed image, such as the maximum error among the errors of each pixel in (t), may be used.

[0042] In S240, the determination unit 130 determines whether or not there is an abnormality in the culture of the cells captured in the captured image, based on the degree of abnormality calculated by the calculation unit 115. The determination unit 130 determines that there is an abnormality in the culture of the cells when the degree of abnormality falls outside the normal range. When using a degree of abnormality whose value increases as the difference between the captured image and the reconstructed image increases, the determination unit 130 may determine that there is an abnormality in the culture of the cells when the degree of abnormality a(t) exceeds a predetermined threshold value.

[0043] When it is determined in S240 that the cell culture is normal, the learning processing unit 150 in the model generation unit 137 adds in S270 the captured image for which it is determined that the cell culture is normal as a new reference image to a set of one or more reference images stored in the reference image storage unit 145. Then, the determination device 10 advances the process to S260.

[0044] When it is determined in S240 that there is an abnormality in the cell culture, the analysis processing unit 132 generates an analysis image in S250, which indicates which area in the captured image contributed to the determination that there was an abnormality in the cell culture. The analysis processing unit 132 may generate an analysis image indicating the part in the captured image that influenced the determination and the degree of influence. In this embodiment, the analysis processing unit 132 generates an analysis image on the captured image, which displays the difference from the reconstructed image. The analysis processing unit 132 may generate an analysis image using a heat map or a contour line. For example, the analysis processing unit 132 reflects the difference from the reconstructed image on the captured image by adding a color corresponding to the magnitude of the difference in pixel value between the captured image and the reconstructed image to the pixel value of the captured image for each pixel of the captured image. Also, for example, the analysis processing unit 132 may add a contour line indicating the difference in pixel value between the captured image and the reconstructed image to the captured image.

[0045] In S260, the result output unit 135 outputs the determination result by the determination unit 130, i.e., whether or not there was an abnormality in the culture of the cells captured in the captured image. Furthermore, when it is determined that there was an abnormality in the culture of the cells, the result output unit 135 outputs the analysis image generated by the analysis processing unit 132.

[0046] According to the above-described determination device 10, by comparing a captured image with a reconstructed image obtained by reconstructing the captured image using a reconstruction model learned using one or more reference images, the degree of abnormality of the captured image based on one or more reference images can be calculated, and it can be determined whether or not there is an abnormality in the cell culture. Furthermore, according to the determination device 10, since it is possible to display on the captured image an area where the difference between the captured image and the reconstructed image is relatively large, it is possible to identify a location to be focused on in the analysis of the captured image determined to have an abnormality in the cell culture.

[0047] 3 shows a learning flow of the determination device 10 according to this embodiment. Prior to the learning process, the reference image storage unit 145 stores one or more reference images of normally cultured cells. Here, the one or more reference images may be images of normally cultured cells from the start to the end of the culture process, or may be images of normally cultured cells from the start to any point before the end of the culture process.

[0048] The reference image storage unit 145 may store one or more reference images selected by the user as images of normally cultured cells before learning as an initial set of reference images. Furthermore, the learning processing unit 150 may select, from among the captured images acquired by the acquisition unit 110, those determined by the determination unit 130 as images of normally cultured cells, and store them in the reference image storage unit 145.

[0049] Between S300 corresponding to the start of the loop and S320 corresponding to the end of the loop, learning processing unit 150 performs a loop process that repeats the processes between S300 and S320 for each reference image stored in reference image storage unit 145. In S310, learning processing unit 150 extracts features of the reference image to be learned, and generates or updates a reconstruction model stored in model storage unit 155 so as to restore the reference image from the extracted features.

[0050] As an example, the reconstruction model may be an autoencoder including an encoder that compresses an input reference image into a feature vector with a smaller number of dimensions and a decoder that restores the reference image from the feature vector, and the encoder and decoder are trained so that the restored reference image more closely resembles the input reference image. The encoder and decoder may include neural networks such as a convolutional neural network (CNN) and a fully connected neural network. The learning processing unit 150 may perform learning processing of the encoder and decoder by an error backpropagation method using the error between the input reference image and the restored reference image.

[0051] The learning processing unit 150 may also use a generative adversarial network (GAN) to compress an input image into a feature vector with a smaller number of dimensions, and generate a reconstruction model that reconstructs an image using the feature vector. For example, in a system including a generator that generates an image reconstructed from a reference image using the reconstruction model, and a discriminator that discriminates whether an image is reconstructed, the learning processing unit 150 performs a learning process for the discriminator so as to improve the discrimination accuracy of the reference image and the image reconstructed from the reference image, and also performs a learning process for the generator so as to reduce the discrimination accuracy by the discriminator. The learning processing unit 150 stores the reconstruction model used in the generator learned in this way in the model storage unit 155.

[0052] In S330, the learning processing unit 150 determines whether the learning process is complete. For example, the learning processing unit 150 determines that the learning process is complete on the condition that the error between an image reconstructed from a reference image using a reconstruction model and the original reference image is less than a predetermined threshold. The learning processing unit 150 may also determine that the learning process is complete in response to the learning process from S300 to S320 being repeated a predetermined number of times. When it is determined that the learning process is not complete, the learning processing unit 150 advances the process to S300 to continue the learning process.

[0053] According to the learning processing unit 150 described above, it is possible to perform a learning process of a reconstruction model that reconstructs a captured image to resemble one or more reference images. Specifically, the learning processing unit 150 trains the reconstruction model to reproduce the features of the reference image as much as possible, but does not train the reconstruction model to reproduce features that are not included in the reference image. Therefore, the reconstruction model reproduces features that are included in the captured image and are similar to the reference image on the reconstructed image, and features that are included in the captured image and are not similar to the reference image can hardly be reproduced on the reconstructed image. Therefore, the reconstruction model can remove or attenuate features in the captured image that are not seen in the reference image obtained by photographing normally cultured cells, and convert the captured image into a reconstructed image that is more similar to the reference image.

[0054] In addition, the learning processing unit 150 may perform online learning of the reconstruction model by performing processing of S310 on the new reference image each time it receives a captured image that is not determined to have had an abnormality in the cell culture as a new reference image.

[0055] The determination device 10 may determine the culture state of cells using time-series images captured at multiple timings, instead of an image captured at a single time t. In this case, the imaging unit 100 stores the images captured in time series during the cell culture process in the captured image storage unit 105 in S200 of Fig. 2, and the acquisition unit 110 acquires the time-series captured images in S210 of Fig. 2.

[0056] The calculation unit 115 may calculate the degree of abnormality of the captured image based on one or more reference images captured in time series during a normal cell culture process. Here, the acquisition unit 110 may use, as the captured images in time series and the reference image, a multi-channel image in which pixel values ​​at each of a plurality of timings are encoded into different channels for each pixel.

[0057] Fig. 4 shows the configuration of a calculation unit 415 according to a first modified example of the present embodiment. In this modified example, the calculation unit 415 is used instead of the calculation unit 115 in the determination device 10 shown in Fig. 1. In this modified example, the components other than the calculation unit 115 in the determination device 10 in Fig. 1 are basically the same, so that a description thereof will be omitted below except for the differences.

[0058] The calculation unit 415 is connected to the acquisition unit 110 and the model generation unit 137. The calculation unit 415 calculates the degree of abnormality of the captured image based on one or more reference images capturing normally cultured cells. The calculation unit 415 has a feature amount calculation unit 420 and an abnormality degree calculation unit 425. The feature amount calculation unit 420 calculates the feature amount of the captured image acquired by the acquisition unit 110.

[0059] The abnormality degree calculation section 425 is connected to the feature amount calculation section 420. The abnormality degree calculation section 425 calculates the abnormality degree based on the feature amount of the captured image and each of the reference feature amounts of one or more reference images. This allows the determination device 10 to determine whether or not there is an abnormality in the culture of cells captured in the captured image based on the difference in the feature amount from one or more reference images capturing normally cultured cells.

[0060] 5 shows the operation of the calculation unit 415 according to a first modified example of the present embodiment. In this modified example, the determination device 10 executes S520 and S530 instead of S220 and S230 in FIG.

[0061] In S520, the feature amount calculation unit 420 calculates the feature amount of the captured image. As an example, the feature amount calculation unit 420 calculates the feature amount obtained by nonlinearly mapping the captured image to a lower dimensional space using a multi-layer perceptron, a CNN, a fully connected neural network, or other neural network. The feature amount calculation unit 420 may calculate a feature amount of a lower dimension than the captured image using a machine learning model other than a neural network. In addition, the feature amount calculation unit 420 may calculate the feature amount of the captured image using a conventional feature amount extraction method in image analysis, such as SURF or AKAZE. The feature amounts shown above may be data in a format different from that of an image including a luminance value for each pixel.

[0062] In S530, the anomaly degree calculation unit 425 calculates the anomaly degree based on the feature amount of the captured image and each of the reference feature amounts of one or more reference images. For example, the anomaly degree calculation unit 425 calculates the degree of difference between the feature amount h(t) of the captured image and an average reference feature amount obtained by averaging the reference feature amounts of one or more reference images as the anomaly degree. In this case, the anomaly degree calculation unit 425 may calculate the anomaly degree a(t) using the following formula (2). Here, μ is the average reference feature amount, and Σ is the covariance of the reference feature amount.

number

[0063] Such an abnormality degree a(t) is calculated by using the difference between the reference feature amounts of the captured image and each of the reference images to determine how much the situation captured in the captured image deviates from one or more reference images. In this way, the abnormality degree calculation unit 425 may calculate the abnormality degree of the captured image using one or more reference images as a reference.

[0064] The abnormality degree calculation unit 425 may calculate the abnormality degree using a calculation method different from the above. For example, the abnormality degree calculation unit 425 may calculate the abnormality degree by using a set of one or more reference feature amounts in one or more reference images, {h iA probability distribution model p(t) of the reference feature is generated from {p(t)} (the subscript i indicates the number of the reference feature), and an anomaly a(t) is calculated based on the likelihood of the feature h(t) of the captured image in the probability distribution model p(t). The anomaly calculation unit 425 may use a negative log-likelihood as the anomaly. The anomaly a(t) calculated in this manner becomes smaller when it is closer to normal (when the likelihood is large) and becomes larger when it is closer to abnormal (when the likelihood is small).

[0065] According to the calculation unit 415 described above, it is possible to calculate the degree of abnormality of the captured image based on the one or more reference images using the feature amounts of the captured image and one or more reference images, and to determine whether or not there is an abnormality in the cell culture. When the calculation unit 415 is used, the model generation unit 137 may generate a feature extraction model used by the feature amount calculation unit 420 using the method exemplified below.

[0066] The learning processing unit 150 in the model generating unit 137 according to this modification may extract features of various captured images of cultured cells, and generate a machine learning model by learning that restores the captured images from the extracted features. The captured images used for this learning may include images of cells with abnormalities in culture. The learning processing unit 150 may then extract a portion of the captured images in this machine learning model that extracts features, and store it in the model storage unit 155 as a feature extraction model. For example, the learning processing unit 150 may generate an autoencoder by learning using various captured images, and extract the encoder portion of the autoencoder as the feature extraction model.

[0067] Alternatively, the determination device 10 may use a trained model of another domain as the feature extraction model, such as a network up to the intermediate feature of a deep learning model created for general image recognition (classification, object detection). The learning processing unit 150 may generate a neural network by learning, which predicts whether or not there is an abnormality in the culture from the captured image, using learning data in which each captured image of a cultured cell is manually labeled as to whether or not there is an abnormality in the culture, and may extract the intermediate stage of this neural network as the feature extraction model. The learning processing unit 150 may also perform a learning process of the feature extraction model using similar learning data to bring the feature amounts output by the feature extraction model according to each captured image associated with normal culture closer to the feature amounts output by the feature extraction model according to each captured image associated with abnormal culture, and to make the feature amounts output by the feature extraction model according to the captured image associated with normal culture and the feature amounts output by the feature extraction model according to the captured image associated with abnormal culture farther apart.

[0068] Furthermore, when the calculation unit 415 is used, the analysis processing unit 132 generates an analysis image that displays the degree of influence of each region on the difference between the feature amount and the reference feature amount on the photographed image. For example, the analysis processing unit 132 may calculate the gradient of the abnormality degree calculation for the photographed image input to the feature amount calculation unit 420, and generate an analysis image in which the region in the photographed image with a larger gradient is emphasized. Furthermore, the analysis processing unit 132 may generate an analysis image for each of a plurality of partial regions in the photographed image based on the difference between the abnormality degree calculated in response to changing the partial region in the photographed image and the abnormality degree calculated from the photographed image. For example, when the abnormality degree calculated from an image in which a certain partial region in the photographed image is masked, for example, becomes a value that is much smaller (a value with a small deviation from the standard) than the abnormality degree calculated from the photographed image itself, the analysis processing unit 132 can determine that the influence of the partial region is large. Alternatively, the analysis processing unit 132 may generate an analysis image in which an area on the captured image that contributes more to the error between the feature amount h(t) of the captured image and the average reference feature amount μ of one or more reference images is emphasized. In this case, the analysis processing unit 132 can use a method such as GradCAM to identify the degree of influence of each area on the difference between the feature amount and the reference feature amount.

[0069] Furthermore, when using time-series captured images, the feature amount calculation section 420 in the calculation section 415 may extract feature amounts from the time-series captured images, and the abnormality degree calculation section 425 in the calculation section 415 may calculate the abnormality degree based on the feature amounts of the time-series captured images and one or more reference feature amounts extracted from one or more time-series reference images. In this case, the feature amount calculation section 420 may extract the feature amounts and reference feature amounts of the time-series captured images and the time-series reference images using, for example, a recurrent neural network (RNN) as a feature extraction model.

[0070] Fig. 6 shows the configuration of a determination device 600 according to a second modified example of this embodiment. The determination device 600 according to this modified example has a configuration in which a preprocessing unit 610 is added to the determination device 10 shown in Fig. 1. Note that components in Fig. 6 that are given the same reference numerals as those in Fig. 1 are the same as those in Fig. 1, and therefore will not be described below except for the differences.

[0071] The determination device 600 further includes a pre-processing unit 610 connected to the acquisition unit 110. The pre-processing unit 610 performs pre-processing on the captured image and supplies it to the calculation unit 115 and the model generation unit 137. When the imaging unit 100 can capture a plurality of captured images while changing the focal position in the depth direction of the field of view, the pre-processing unit 610 may perform a three-dimensional reconstruction process of an image of a cell or a cell cluster (such as a colony, a spheroid, or an organoid) using these multiple captured images. That is, the pre-processing unit 610 may reconstruct a three-dimensional captured image in which the cell or cell cluster is converted into three-dimensional data from the multiple captured images sliced ​​in the depth direction. This allows the calculation unit 115 to calculate the degree of abnormality using the three-dimensional captured images.

[0072] The pre-processing unit 610 may also perform edge enhancement processing on the image of the cell or cell aggregate. For example, the pre-processing unit 610 may output a captured image obtained by the acquisition unit 110, in which three-dimensional information is convolved with two-dimensional information using a phase contrast method or the like to enhance edges. The model generating unit 137 of the determination device 600 may use a reference image that has been pre-processed in the same manner as the pre-processing unit 610, as each of one or more reference images to be stored in the reference image storage unit 145. The pre-processing unit 610 may also be applied to a configuration in which the calculation unit 415 is used instead of the calculation unit 115 in FIG. 1.

[0073] Fig. 7 shows the configuration of a determination device 700 according to a third modified example of this embodiment. The determination device 700 according to this modified example is a modification of the determination device 10 in Fig. 1 so as to perform determination taking into account the culture conditions and imaging conditions of the cells. In Fig. 6, the components with the same reference numerals as in Fig. 1 are the same as those in Fig. 1 except that at least one of the culture conditions or the imaging conditions is used, and therefore, description thereof will be omitted below except for the differences.

[0074] The determination device 700 further includes a culture unit 710. The culture unit 710 inputs cell culture conditions and cultures the cells in the measurement target 20 according to the culture conditions. The cell culture conditions may be represented by a set of at least one of environmental parameters such as temperature, operation parameters such as drug administration during the culture process, or other culture-related parameters.

[0075] In this modification, the imaging unit 100 inputs the imaging conditions of the measurement target 20 and images the measurement target 20 according to the imaging conditions. The imaging conditions may be represented by a set of at least one of the parameters such as lens magnification, exposure time, illumination, imaging position (X, Y, Z coordinates, etc.), or other parameters that may cause differences in the captured image even when the same subject is imaged.

[0076] The photographed image storage unit 105 stores the photographed image in association with the culture conditions and the photographing conditions. The acquisition unit 110 acquires the culture conditions and the photographing conditions related to the photographed image to be processed in addition to the photographed image. The calculation unit 115 calculates the degree of abnormality based on one or more reference images of cells normally cultured under culture conditions corresponding to the acquired culture conditions and photographed under the photographing conditions corresponding to the acquired photographing conditions. In this case, the model storage unit 155 stores a plurality of reconstructed models for each culture condition and photographing condition, and the calculation unit 115 may reconstruct the photographed image using a reconstructed model corresponding to the culture conditions and the photographing conditions related to the photographed image to be processed among the plurality of reconstructed models stored in the model storage unit 155. In addition, the abnormality degree calculation unit 125 may switch a threshold value used to determine whether or not there is an abnormality in the cell culture according to the culture conditions and the photographing conditions. The learning processing unit 150 may perform a learning process of the reconstructed model using one or more reference images for each culture condition and photographing condition.

[0077] According to this modification, the determination device 700 can perform reconstruction processing and abnormality degree calculation suitable for the culture conditions and imaging conditions related to the captured images for the captured images obtained under various culture conditions and imaging conditions. This allows the determination device 700 to appropriately determine whether or not there is an abnormality in the cell culture based on the captured images obtained under various culture conditions and imaging conditions.

[0078] The determination device 700 may perform the determination by taking into consideration only one of the culture conditions and the image capturing conditions. The determination device 700 may further include a calculation unit 415 instead of the calculation unit 115.

[0079] Fig. 8 shows the configuration of a determination device 800 according to a fourth modified example of this embodiment. The determination device 800 according to this modified example is a modification of the determination device 10 in Fig. 1, which is modified so as to perform determination using images captured at multiple locations. In Fig. 8, components with the same reference numerals as in Fig. 1 are the same as those in Fig. 1, except that at least one of the culture conditions or the image capturing conditions is used, and therefore, description thereof will be omitted below except for the differences.

[0080] In this modification, the acquisition unit 110 acquires a captured image for each of the multiple fields of view. The acquisition unit 110 may acquire partial captured images corresponding to each of the multiple fields of view at once by acquiring one captured image including the multiple fields of view. Alternatively, the acquisition unit 110 may acquire multiple captured images obtained by capturing each of the multiple fields of view separately.

[0081] Here, the acquiring unit 110 may acquire a plurality of captured images of a plurality of locations within a culture unit such as a plate or a well within a plate used for culturing the measurement target 20. Alternatively, the acquiring unit 110 may acquire each captured image by capturing a plurality of plates or a plurality of wells within the field of view.

[0082] The determination device 800 includes a distribution unit 810 and a plurality of calculation units 815-1 to N (also referred to as "calculation units 815") instead of the calculation unit 115 in Fig. 1. The distribution unit 810 distributes captured images of a plurality of fields of view to the plurality of calculation units 815-1 to N. In the example of this figure, the determination device 800 includes N calculation units 815.

[0083] Each of the calculation units 815-1 to N is connected to the distribution unit 810. Each calculation unit 815 calculates the degree of abnormality of the captured image received from the distribution unit 810. Each calculation unit 815 may have a similar configuration to the calculation unit 115 in FIG. 1, and may calculate the degree of abnormality of the captured image in the same manner as the calculation unit 115. Alternatively, each calculation unit 815 may have a similar configuration to the calculation unit 415 in FIG. 4, and may calculate the degree of abnormality of the captured image in the same manner as the calculation unit 415.

[0084] The determination unit 130 is connected to each of the multiple calculation units 815-1 to N. The determination unit 130 outputs a determination result that integrates the degrees of abnormality of the captured images in the multiple fields of view calculated by each of the multiple calculation units 815.

[0085] In this modification, the model generating unit 137 is connected to the distribution unit 810 and the multiple calculation units 815-1 to N. The learning processing unit 150 in the model generating unit 137 receives the multiple captured images from the calculation unit 815. The learning processing unit 150 receives the degree of abnormality of the captured image processed by each calculation unit 815 from each calculation unit 815, and adds the captured image to the reference image storage unit 145 as a new reference image on the condition that the culture of the cells captured in the captured image is normal. Then, the learning processing unit 150 performs learning processing of the reconstruction model or the feature extraction model using the one or more reference images to which the new reference image has been added. The reconstruction model or the feature extraction model generated by the learning processing unit 150 is used by each of the multiple calculation units 815-1 to N.

[0086] Fig. 9 shows a determination flow of a determination device 800 according to a fourth modified example of this embodiment. In the determination flow of this modified example, the operations of blocks having the same step numbers as those in Fig. 2 are similar to the operations of the corresponding blocks in Fig. 2, and therefore will not be described below except for the differences.

[0087] In S200, the imaging unit 100 captures a captured image for each of the multiple fields of view. In S210, the acquisition unit 110 acquires a captured image for each of the multiple fields of view. Here, the acquisition unit 110 may acquire a captured image including multiple fields of view for each culture plate or well. Alternatively, the acquisition unit 110 may acquire a captured image including each of the multiple fields of view for each of the multiple culture plates or wells.

[0088] In S915, the distribution unit 810 distributes the captured images of the multiple fields of view acquired by the acquisition unit 110 to the multiple calculation units 815. Here, when a single captured image including multiple fields of view is received, the distribution unit 810 cuts out partial captured images for each of the multiple fields of view from the received captured image and distributes them to each of the multiple calculation units 815-1 to N. When an individual captured image for each of the multiple fields of view is received, the distribution unit 810 distributes each of the multiple captured images to each of the multiple calculation units 815-1 to N.

[0089] Between S917 corresponding to the beginning of the loop and S945 corresponding to the end of the loop, the calculation units 815-1 to N perform a loop process of executing the process between S917 and S945 for each captured image distributed by the distribution unit 810. In this modification, in S940 and S270, the learning processing unit 150 in the model generation unit 137 adds a captured image determined to have been cultured as normal using the degree of abnormality ("Y" in S940) to the reference image storage unit 145 as a new reference image (S270). That is, in this modification, when the learning processing unit 150 determines that the cultured cells were normal in some of the captured images and determines that the cultured cells were abnormal in the remaining captured images, the learning processing unit 150 uses the captured images determined to have been cultured as normal individually as new reference images even if the determination unit 130 determines that the cultured cells were abnormal as a whole.

[0090] In S947, the determination unit 130 outputs a determination result that integrates the degrees of abnormality of the captured images of the multiple fields of view calculated by each of the multiple calculation units 815. For example, when the determination unit 130 uses a degree of abnormality that increases as the cell culture becomes more abnormal, the determination unit 130 outputs a determination result that there is an abnormality in the cell culture when the degree of abnormality of at least one captured image exceeds a threshold. Alternatively, the determination unit 130 may output a determination result that there is an abnormality in the cell culture when the degrees of abnormality of more than a predetermined number of captured images exceed a threshold.

[0091] When the determination result that there was an abnormality in the cell culture is output ("N" in S240), the analysis processing unit 132 generates an analysis image indicating which area in each captured image contributed to the determination that there was an abnormality in the cell culture in S250. In S260, the result output unit 135 outputs the integrated determination result by the determination unit 130. Furthermore, when it is determined that there was an abnormality in the cell culture, the result output unit 135 outputs each analysis image generated by the analysis processing unit 132 and corresponding to each captured image.

[0092] According to the above-described determination device 800, it is possible to comprehensively determine whether or not there is an abnormality in the cell culture by using captured images of multiple fields of view for each culture plate or well, or for multiple culture plates or multiple wells, etc. As a result, the determination device 800 can make a rational determination by using captured images of multiple fields of view even when there is a statistically certain probability that an abnormality will occur in the culture of some cells.

[0093] Fig. 10 shows the configuration of a determination device 1000 according to a fifth modified example of this embodiment. The determination device 800 according to this modified example is a modification of the determination device 10 in Fig. 1, which is modified so as to perform determination using a plurality of types of models. In Fig. 10, components with the same reference numerals as in Fig. 1 are basically the same as those in Fig. 1, and therefore will not be described below except for the differences.

[0094] The determination device 1000 includes a plurality of calculation units 1015-1 to M (also referred to as "calculation units 1015") instead of the calculation unit 115 in Fig. 1. In the example of this figure, the determination device 1000 includes M calculation units 1015.

[0095] The multiple calculation units 1015-1 to M are connected to the acquisition unit 110. Each calculation unit 1015 receives the same captured image acquired by the acquisition unit 110. Then, each calculation unit 1015 calculates the degree of abnormality of the captured image using a model provided to each calculation unit 1015.

[0096] Each calculation unit 1015 may have the same configuration as the calculation unit 115 in Fig. 1. In this case, the multiple reconstructed image generation units 120 included in the multiple calculation units 1015 input the captured image to multiple different reconstruction models, respectively, to generate multiple reconstructed images. The multiple abnormality calculation units 125 included in the multiple calculation units 1015 calculate the abnormality degree for each of the multiple reconstructed images.

[0097] Further, each calculation unit 1015 may have the same configuration as the calculation unit 415 in FIG. 4. In this case, the plurality of feature amount calculation units 420 included in the plurality of calculation units 1015 input the captured image to each of the plurality of feature extraction models different from each other to calculate the plurality of feature amounts. Each of the plurality of abnormality degree calculation units 425 included in the plurality of calculation units 1015 calculates the abnormality degree based on the feature amount calculated from the captured image using the feature extraction models different from each other and the reference feature amount of one or more reference images calculated using the feature extraction models. Note that some of the plurality of calculation units 1015 may have the same configuration as the calculation unit 115 in FIG. 1, and the other part may have the same configuration as the calculation unit 415 in FIG. 4.

[0098] The determination unit 130 is connected to each of the multiple calculation units 1015-1 to M. The determination unit 130 outputs a determination result that integrates the degrees of abnormality for each model calculated by each of the multiple calculation units 1015.

[0099] Moreover, the determination device 1000 may include a plurality of model generation units 1037-1 to M (also indicated as "determination devices 1037") instead of the model generation unit 137 in FIG. 1. Each determination device 1037 adds the captured image to the reference image storage unit 145 in each determination device 1037 as a new reference image on the condition that the culture of the cells captured in the captured image is determined to be normal. Then, the learning processing unit 150 in each determination device 1037 performs learning processing of the reconstruction model or the feature extraction model using one or more reference images to which the new reference image has been added. In this modification, the multiple models generated by the multiple determination devices 1037 are used by the multiple calculation units 1015-1 to M, respectively.

[0100] Here, the multiple reconstruction models or feature extraction models may differ from each other due to different machine learning algorithms, such as a model using an autoencoder, a model using a generative adversarial network, or a machine learning model other than a neural network, such as an SVM or a statistical learning model. The multiple reconstruction models or feature extraction models may differ from each other due to different machine learning parameters, such as the number of layers in a neural network or the number of neurons per layer. The multiple reconstruction models or feature extraction models may also differ in how they are learned, such as by using different sets of training data (e.g., sets of reference images used for learning) or by using different learning intensities.

[0101] Fig. 11 shows a determination flow of a determination device 1000 according to a fifth modified example of this embodiment. In the determination flow of this modified example, the operations of blocks having the same step numbers as those in Fig. 2 are similar to the operations of the corresponding blocks in Fig. 2, and therefore will not be described below except for the differences.

[0102] In S200, the imaging unit 100 captures an image of a field of view including cells, and stores the captured image in the captured image storage unit 105. In S210, the acquisition unit 110 acquires the captured image from the captured image storage unit 105.

[0103] The determination device 1000 performs a loop process in which each process between S1117 corresponding to the beginning of the loop and S1145 corresponding to the end of the loop is executed for each of a plurality of different feature extraction models. In the loop process, the plurality of calculation units 1015-1 to 1015-M executes the same processes as S220 and S230 in FIG. 2 or S520 and S530 in FIG.

[0104] In S1147, the determination unit 130 outputs a determination result that integrates the multiple abnormality degrees output by the multiple calculation units 1015. The determination unit 130 may calculate a determination result as to whether or not there is an abnormality in the cell culture, using statistics of the multiple abnormality degrees. The determination unit 130 may use at least one of the average value, median value, maximum value, minimum value, variance, or other values ​​as such statistics. For example, the determination unit 130 determines that there is an abnormality in the cell culture when the average value of the multiple abnormality degrees exceeds a predetermined threshold value.

[0105] Furthermore, the judgment unit 130 may use the abnormality degree output by each calculation unit 1015 to calculate the individual judgment result by each calculation unit 1015, and determine the final judgment result by majority vote or the like. For example, when the judgment unit 130 uses the abnormality degree that becomes a large value according to the abnormality of the cell culture, if the abnormality degree output by each calculation unit 1015 exceeds a predetermined threshold, the judgment unit 130 judges that there is an abnormality in the cell culture as a result of the calculation unit 1015 alone. Then, the judgment unit 130 determines the judgment result that there is an abnormality in the cell culture by taking a majority vote of the judgment results based on the abnormality degrees of each calculation unit 1015, or by a condition that the abnormality of the cell culture is judged to be abnormal for the abnormality degrees exceeding a predetermined number, or the like.

[0106] The determination device 1000 performs the processes of S240 to S270 in FIG. 11 in the same manner as S240 to S270 in FIG. 2. Here, the analysis processing unit 132 may generate an analysis image displaying a distribution of the degree of influence on the degree of abnormality output by each calculation unit 1015 on the photographed image in S250. For example, the analysis processing unit 132 may generate an analysis image on the photographed image by combining the difference between the reconstructed image generated by the reconstructed image generating unit 120 in each calculation unit 1015 and the photographed image. In this case, the analysis processing unit 132 may reflect the difference between the reconstructed image and the photographed image in the analysis image on the condition that the degree of abnormality output by the calculation unit 1015 exceeds a predetermined threshold value. Thereby, the analysis processing unit 132 can indicate the area in the photographed image that has influenced the degree of abnormality when it should be determined that there is an abnormality in the cell culture based on the degree of abnormality of the calculation unit 1015 alone, for example.

[0107] According to the above-described determination device 1000, it is possible to determine whether or not an abnormality has occurred in cell culture by using a plurality of models. This allows the determination device 1000 to further improve the accuracy of determination.

[0108] In addition, some models for analyzing captured images perform probabilistic output. When at least one of the multiple calculation units 1015 uses such a probabilistic model, the determination device 1000 may perform multiple trials of evaluation of the captured image by the calculation unit 1015 using the probabilistic model, and determine the determination result using the statistics of the abnormality degree obtained as a result of the multiple trials.

[0109] Various embodiments of the present invention may be described with reference to flow charts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and / or a processor provided with computer readable instructions stored on a computer readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0110] A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0111] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0112] Computer readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus, such as a general purpose computer, special purpose computer, or other computer, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, etc., to execute the computer readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0113] 12 illustrates an example of a computer 2200 in which aspects of the present invention may be embodied in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to an embodiment of the present invention and / or to perform a process or steps of a process according to an embodiment of the present invention. Such programs may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0114] The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are connected to each other by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0115] The CPU 2212 operates according to a program stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphic controller 2216 retrieves image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into the graphic controller 2216 itself, and causes the image data to be displayed on the display device 2218.

[0116] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0117] The ROM 2230 stores therein a boot program or the like executed by the computer 2200 upon activation, and / or a program that depends on the hardware of the computer 2200. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0118] The programs are provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 2224, the RAM 2214, or the ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. Information processing described in these programs is read by the computer 2200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing information manipulation or processing according to the use of the computer 2200.

[0119] For example, when communication is performed between the computer 2200 and an external device, the CPU 2212 may execute a communication program loaded in the RAM 2214 and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2212, the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0120] Furthermore, the CPU 2212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.

[0121] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 2212 may perform various types of processing on the data read from the RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 2214. The CPU 2212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2212 may search for an entry that matches a condition, in which an attribute value of the first attribute is specified, from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0122] The above-described programs or software modules may be stored in a computer-readable medium on the computer 2200 or in the vicinity of the computer 2200. Also, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the programs to the computer 2200 via the network.

[0123] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0124] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0125] 10 judgment device, 20 measurement object, 100 imaging unit, 105 captured image storage unit, 110 acquisition unit, 115 calculation unit, 120 reconstructed image generation unit, 125 abnormality degree calculation unit, 130 judgment unit, 132 analysis processing unit, 135 result output unit, 137 model generation unit, 145 reference image storage unit, 150 learning processing unit, 155 model storage unit, 415 calculation unit, 420 feature amount calculation unit, 425 abnormality degree calculation unit, 600 judgment device, 610 preprocessing unit, 700 judgment device, 710 culture unit, 800 judgment device, 810 distribution unit, 815-1 to N calculation unit, 1000 judgment device, 1015-1 to M calculation unit, 1037-1 to M model generation unit, 2200 computer, 2201 DVD-ROM, 2210 host controller, 2212 CPU, 2214 RAM, 2216 graphics controller, 2218 display device, 2220 input / output controller, 2222 communication interface, 2224 hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 input / output chip, 2242 keyboard

Claims

1. an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; Equipped with The calculation unit is a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image; having the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image; The present invention further includes a learning processing unit that performs learning processing of the reconstructed model using the one or more reference images obtained by photographing a field of view including cells that have been determined to have been normally cultured. Judgment device.

2. The determination device according to claim 1, wherein the learning processing unit adds the captured image for which the determination unit has not determined that there was an abnormality in cell culture to the set of one or more reference images as a new reference image, and performs a learning process for the reconstruction model.

3. an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; Equipped with The calculation unit is a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image; having the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image; A plurality of the calculation units are provided, The plurality of reconstructed image generating units included in the plurality of calculation units input the captured image to each of the plurality of reconstructed models different from each other to generate a plurality of reconstructed images; The plurality of abnormality calculation units included in the plurality of calculation units calculate the abnormality degree for each of the plurality of reconstructed images, The determination unit outputs a determination result obtained by integrating the degree of abnormality of each of the plurality of reconstructed images. Judgment device.

4. an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; Equipped with The acquisition unit acquires the captured image for each of a plurality of fields of view, A plurality of the calculation units are provided, The plurality of calculation units calculate the degree of abnormality of the captured image of each of the plurality of fields of view, The determination unit outputs a determination result obtained by integrating the degree of abnormality of the captured images of the plurality of fields of view. Judgment device.

5. The determination device according to claim 4 , wherein the determination unit outputs a determination result that integrates the degree of abnormality for each culture plate or each well.

6. The determination device according to claim 4 , wherein the determination unit outputs a determination result that integrates the degrees of abnormality for a plurality of culture plates or wells.

7. an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; Equipped with The acquisition unit further acquires cell culture conditions, The calculation unit calculates the degree of abnormality based on the one or more reference images obtained by photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions. Judgment device.

8. The calculation unit is a feature amount calculation unit that calculates a feature amount of the captured image; an anomaly degree calculation unit that calculates the anomaly degree based on a feature amount of the captured image and each of the reference feature amounts of the one or more reference images; The determination device according to claim 1 , further comprising:

9. The acquisition unit acquires the captured images of a cell culture process in time series, The calculation unit calculates the degree of abnormality based on the one or more reference images obtained by capturing a normal cell culture process in time series. The determination device according to claim 1 .

10. A pre-processing unit that performs pre-processing on the captured image is further provided, The calculation unit calculates the degree of abnormality using the captured image that has been preprocessed. The determination device according to claim 1 .

11. The determination device according to claim 10 , wherein the pre-processing includes at least one of a three-dimensional reconstruction process and an edge enhancement process of an image of a cell or a cell cluster.

12. The determination device acquires a captured image of a field of view including cells; The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality. Including, Calculating the degree of abnormality of the captured image The determination device generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; the determination device calculates the degree of abnormality based on a difference between the photographed image and the reconstructed image; Including, In generating the reconstructed image, the determination device extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model that has been trained to restore each reference image from the extracted features, thereby generating the reconstructed image; The determination device further includes performing a learning process of the reconstructed model using the one or more reference images obtained by capturing a field of view including cells determined to have been normally cultured. Judgment method.

13. The determination device acquires a captured image of a field of view including cells; The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality. Including, Calculating the degree of abnormality of the captured image The determination device generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; the determination device calculates the degree of abnormality based on a difference between the photographed image and the reconstructed image; Including, In generating the reconstructed images, the determination device extracts features of each of the one or more reference images, and inputs the captured images into a plurality of different reconstruction models that are trained to restore each of the reference images from the extracted features, thereby generating a plurality of the reconstructed images; In determining whether or not an abnormality has occurred in the cell culture, the determination device outputs a determination result that integrates the degree of abnormality of each of the plurality of reconstructed images. Judgment method.

14. The determination device acquires a captured image of a field of view including cells; The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality. Including, In acquiring the captured images, the determination device acquires the captured images for each of a plurality of fields of view, In calculating the degree of abnormality of the captured image, the determination device calculates the degree of abnormality of each of the captured images in the plurality of fields of view, In determining whether or not there is an abnormality in the cell culture, the determination device outputs a determination result that integrates the degree of abnormality of the captured images of the plurality of fields of view. Judgment method.

15. The determination device acquires a captured image of a field of view including cells; The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality. Including, The determination device further acquires cell culture conditions, In calculating the degree of abnormality of the captured image, the determination device calculates the degree of abnormality based on the one or more reference images obtained by photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions. Judgment method.

16. The method is executed by a computer, causing the computer to: an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; and make it work. The calculation unit is a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image; having the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image; The computer, a learning processing unit that performs learning processing of the reconstructed model using the one or more reference images obtained by photographing a field of view including cells that have been determined to have been normally cultured; A judgment program that further functions as a.

17. The method is executed by a computer, causing the computer to: an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; and make it work. The calculation unit is a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images; an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image; having the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image; causing the computer to function as a plurality of the calculation units; The plurality of reconstructed image generating units included in the plurality of calculation units input the captured image to each of the plurality of reconstructed models different from each other to generate a plurality of reconstructed images; The plurality of abnormality calculation units included in the plurality of calculation units calculate the abnormality degree for each of the plurality of reconstructed images, The determination unit outputs a determination result obtained by integrating the degree of abnormality of each of the plurality of reconstructed images. Judging program.

18. The method is executed by a computer, causing the computer to: an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; and make it work. The acquisition unit acquires the captured image for each of a plurality of fields of view, causing the computer to function as a plurality of the calculation units; The plurality of calculation units calculate the degree of abnormality of the captured image of each of the plurality of fields of view, The determination unit outputs a determination result obtained by integrating the degree of abnormality of the captured images of the plurality of fields of view. Judging program.

19. The method is executed by a computer, causing the computer to: an acquisition unit for acquiring a captured image of a field of view including cells; A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells; a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality; and make it work. The acquisition unit further acquires cell culture conditions, The calculation unit calculates the degree of abnormality based on the one or more reference images obtained by photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions. Judging program.

Images