Cell culture evaluation device, method of operating the cell culture evaluation device, operating program of the cell culture evaluation device

Abstract

Provided is a cell culture evaluation device equipped with at least one processor. The processor is configured to: acquire cell images obtained by taking images of cells being cultured; input the cell images to an image machine learning model; cause the image machine learning model to output an image feature amount set comprising a plurality of types of image feature amounts regarding the cell images; input the image feature amount set to a data machine learning model; and cause the data machine learning model to output an expression level set comprising a plurality of types of ribonucleic acid expression levels of the cells.

Description

Technical Field

[0001] This application claims the priority of U.S. Provisional Application No. 63 / 002696, filed on March 31, 2020, the entire content of which is incorporated herein by reference. The technology disclosed herein relates to a cell culture evaluation device, a method for operating the cell culture evaluation device, and an operation program for the cell culture evaluation device.

Background Art

[0002] In the field of culturing cells such as induced pluripotent stem (iPS) cells, a technology has been proposed that enables a computer to predict the future quality of cells when culturing progresses from the current time based on cell images obtained by photographing the current cells. For example, Japanese Patent Application Laid-Open No. 2009-044974 describes a technology for deriving a predicted value of cell quality from a plurality of types of index values related to the morphology of cells.

[0003] In the technology described in Japanese Patent Application Laid-Open No. 2009-044974, first, a cell image is input into a commercially available image analysis software designed in advance to obtain a plurality of types of predetermined index values. Then, the obtained plurality of types of index values are input into a machine learning model using a fuzzy neural network, and a predicted value is output from the machine learning model. Examples of the index values include the area, length, circularity, ellipticity, radii of the inscribed circle and circumscribed circle of the cell, etc. Examples of the predicted values include the cell growth rate, remaining mitotic time, differentiation degree, carcinogenicity degree, etc. The machine learning model is a model that is learned by providing, as learning data, a combination of a plurality of types of index values obtained from a certain cell image and the measured value of the quality of the cells shown in the cell image.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the field of cell culture, the expression level of ribonucleic acid (RNA) is crucial information for predicting cell quality, such as differentiation and carcinogenicity. Currently, research is actively underway to investigate the causal relationship between expression levels and cell quality, and it is becoming clear that the expression level of certain specific ribonucleic acids (called markers) has a significant impact on cell quality. However, much of the causal relationship between expression levels and cell quality remains unclear. Therefore, in order to further accelerate the elucidation of the causal relationship between expression levels and cell quality, there has been a demand to predict the expression levels of multiple types of ribonucleic acids, not limited to markers, that are thought to be the basis for predicting cell quality.

[0005] One possible method for predicting expression levels is to mimic the technology described in Japanese Patent Publication No. 2009-044974, as follows: Cell images are input into commercially available image analysis software to obtain a set of predetermined indicator values, as exemplified above. These obtained indicator values ​​are then input into a machine learning model, which outputs the expression levels of multiple ribonucleic acids. However, these indicator values ​​are limited to those that are intuitively perceptible to the human eye, such as cell area, length, and roundness, and are arbitrarily set by humans. Such limited indicator values ​​are considered unsuitable for predicting the expression levels of multiple ribonucleic acids, a method that has not yet been implemented.

[0006] The technology disclosed herein provides a cell culture evaluation device capable of appropriately predicting the expression levels of multiple types of ribonucleic acids in cells, a method for operating the cell culture evaluation device, and an operating program for the cell culture evaluation device. [Means for solving the problem]

[0007] A first aspect of this disclosure is a cell culture evaluation device comprising at least one processor, the processor acquires a cell image obtained by photographing cells in culture, inputs the cell image into an image machine learning model, causes the image machine learning model to output an image feature set consisting of multiple types of image features related to the cell image, inputs the image feature set into a data machine learning model, and causes the data machine learning model to output an expression level set consisting of the expression levels of multiple types of ribonucleic acids of cells.

[0008] The processor may also have control over the display of expression level sets.

[0009] The processor may acquire multiple cell images obtained by taking multiple images of a single culture vessel in which cells are cultured, input each of the multiple cell images into an image machine learning model, and have the image machine learning model output a set of image features for each of the multiple cell images.

[0010] The processor may aggregate the multiple image feature sets output for each of the multiple cell images into a number that can be handled by the machine learning model for the data, input the aggregated image feature sets into the machine learning model for the data, and have the machine learning model for the data output an expression level set for each of the aggregated image feature sets.

[0011] Multiple cell images may include at least one of the following: cell images obtained using different imaging methods, and cell images obtained by imaging cells stained with different dyes.

[0012] In addition to the image feature set, the processor may also input reference information to the machine learning model for the data that serves as a guide for the output of the expression level set.

[0013] Reference information may include information related to cell morphology and information on the components of the cell culture supernatant.

[0014] Morphology-related information may include at least one of the following: cell type, donor, confluence, quality, and reprogramming method.

[0015] An autoencoder having a compression unit that converts a cell image into a set of image features and a restoration unit that generates a reconstructed image of the cell image from the set of image features may be used as an image machine learning model.

[0016] The compression unit may consist of multiple extraction units, each prepared according to the size of the target group to be extracted within the cell image. Each extraction unit may have a convolutional layer to extract a set of target group features, each consisting of multiple types of target group-specific features, and a fully connected layer to convert the multiple target group feature sets output from the multiple extraction units into an image feature set.

[0017] The autoencoder may be trained using a generative adversarial network that has a discriminator that determines whether the cell image and the reconstructed image are identical.

[0018] The autoencoder may be trained by inputting cell morphology-related information into the reconstruction unit, in addition to the image feature set from the compression unit.

[0019] Morphology-related information may include at least one of the following: cell type, donor, confluence, quality, and reprogramming method.

[0020] A compression unit in a convolutional neural network that converts cell images into a set of image features and an output unit that outputs evaluation labels about the cells based on the set of image features may be used as a machine learning model for images.

[0021] A second aspect of the present disclosure is a method of operating a cell culture evaluation device, which includes acquiring a cell image obtained by photographing cells in culture, inputting the cell image into an image machine learning model, and performing a first process of outputting a set of image feature amounts composed of a plurality of types of image feature amounts related to the cell image from the image machine learning model, inputting the set of image feature amounts into a data machine learning model, and performing a second process of outputting a set of expression amounts composed of expression amounts of a plurality of types of ribonucleic acids of the cells from the data machine learning model, and the processor executes a process including this.

[0022] A third aspect of the present disclosure is an operation program of a cell culture evaluation device, which functions as an acquisition unit that acquires a cell image obtained by photographing cells in culture, a first processing unit that inputs the cell image into an image machine learning model and outputs a set of image feature amounts composed of a plurality of types of image feature amounts related to the cell image from the image machine learning model, and a second processing unit that inputs the set of image feature amounts into a data machine learning model and outputs a set of expression amounts composed of expression amounts of a plurality of types of ribonucleic acids of the cells from the data machine learning model, to cause the processor of the cell culture evaluation device to function.

Effect of the Invention

[0023] According to the technology of the present disclosure, it is possible to provide a cell culture evaluation device, a method of operating a cell culture evaluation device, and an operation program of a cell culture evaluation device that can appropriately predict the expression amounts of a plurality of types of ribonucleic acids of cells.

Brief Description of the Drawings

[0024] [Figure 1] It is a diagram showing a cell culture evaluation device and the like. [Figure 2] It is a diagram showing a plurality of regions obtained by dividing a well. [Figure 3] It is a block diagram showing a computer constituting a cell culture evaluation device. [Figure 4] It is a block diagram showing a processing unit of a CPU of a cell culture evaluation device. [Figure 5] It is a diagram showing the processing of the first processing unit. [Figure 6]This diagram shows the processing in the aggregation section. [Figure 7] This diagram shows the processing of the second processing unit. [Figure 8] This is a diagram showing the specified screen. [Figure 9] This is a diagram showing the results display screen. [Figure 10] This figure shows the configuration of the autoencoder and the structure of the image model. [Figure 11] This figure shows an overview of the processing during the learning phase of an autoencoder. [Figure 12] This is a diagram showing the compression section. [Figure 13] This diagram shows the groups of cells within the cell image that each extraction unit in the compression section is responsible for extracting. [Figure 14] This diagram shows the details of the extraction and binding sections. [Figure 15] This diagram shows the details of the extraction section. [Figure 16] This is an explanatory diagram of the convolution process. [Figure 17] This figure shows how the feature maps for each target group are constructed. [Figure 18] This is an explanatory diagram that applies the concept of a convolutional neural network to a convolutional processing operation using filters. [Figure 19] This is an explanatory diagram of the maximum value pooling process. [Figure 20] This is an explanatory diagram illustrating the concept of a fully connected layer. [Figure 21] This figure shows an overview of the processing during the training phase of the data model. [Figure 22] This flowchart shows the processing procedure for the cell culture evaluation device. [Figure 23] This figure shows an overview of the processing during the learning phase of an autoencoder using a generative adversarial network. [Figure 24] This figure shows how an autoencoder is incorporated as a generator in a generative adversarial network, and how the discriminator of the generative adversarial network determines whether the training cell image and the training reconstructed image are identical. [Figure 25A] This figure shows the amount of autoencoder update in response to the discriminator's discrimination result, specifically the case where the discriminator determined that the training cell image and the reconstructed training image were not identical. [Figure 25B] This figure shows the amount of autoencoder update in response to the discriminator's discrimination result, specifically the case where the training cell image and the reconstructed training image were determined to be identical. [Figure 26] This figure shows a third embodiment in which, during the learning phase of the autoencoder, cell morphology-related information is input to the reconstruction unit in addition to the image feature set from the compression unit. [Figure 27] This is a diagram showing morphological information. [Figure 28] This figure shows a fourth embodiment in which reference information is input into the data model in addition to the image feature set. [Figure 29] This is a diagram showing the components of the culture supernatant. [Figure 30] This figure shows an overview of the processing during the training phase of the data model shown in Figure 28. [Figure 31] This figure shows a fifth embodiment in which the compression section of a convolutional neural network is used as an image model. [Figure 32] This figure shows an overview of the processing during the learning phase of the convolutional neural network shown in Figure 31. [Figure 33] This figure shows an example of cell images obtained by capturing multiple cell images using different imaging methods. [Figure 34] This figure shows an example of using multiple cell images, each obtained by photographing cells stained with a different dye. [Modes for carrying out the invention]

[0025] [First Embodiment] In Figure 1, the cell culture evaluation device 10 is, for example, a desktop personal computer, which receives cell images 12 from the imaging device 11. The imaging device 11 is, for example, a phase-contrast microscope. The imaging device 11 is fitted with a well plate 15 in which a plurality of wells 14 for culturing cells 13 are formed. The imaging device 11 takes a picture of the cells 13 being cultured in one well 14 to obtain a cell image 12. The imaging device 11 transmits the cell image 12 to the cell culture evaluation device 10. The cells 13 are, for example, iPS cells. Note that the well 14 is an example of a "culture vessel" according to the technology of this disclosure.

[0026] As shown in Figure 2, the imaging device 11 captures multiple cell images 12 for each of the multiple regions 18 into which the well 14 is divided. For this reason, as indicated by the arrows in Figure 1, the imaging device 11 is movable in two mutually orthogonal directions.

[0027] As shown in Figure 3, the computer constituting the cell culture evaluation apparatus 10 includes a storage device 20, memory 21, CPU (Central Processing Unit) 22, communication unit 23, display 24, and input device 25. These are interconnected via a bus line 26.

[0028] The storage device 20 is a hard disk drive built into the computer constituting the cell culture evaluation apparatus 10, or connected via a cable or network. Alternatively, the storage device 20 is a disk array consisting of multiple hard disk drives installed in series. The storage device 20 stores control programs such as the operating system, various application programs, and various data associated with these programs. A solid-state drive may be used instead of a hard disk drive.

[0029] Memory 21 is work memory for the CPU 22 to execute processing. The CPU 22 loads programs stored in storage device 20 into memory 21 and executes processing according to the programs. In this way, the CPU 22 comprehensively controls all parts of the computer.

[0030] The communication unit 23 controls the transmission of various information to external devices such as the imaging device 11. The display 24 displays various screens. The computer constituting the cell culture evaluation device 10 receives operation instructions from the input device 25 through the various screens. The input device 25 includes a keyboard, mouse, touch panel, etc.

[0031] As shown in Figure 4, the storage device 20 of the cell culture evaluation device 10 stores an operating program 30. The operating program 30 is an application program that causes the computer to function as the cell culture evaluation device 10. In other words, the operating program 30 is an example of an "operating program for a cell culture evaluation device" related to the technology of this disclosure. The storage device 20 also stores a group of cell images 35, a machine learning model for images (hereinafter abbreviated as the image model) 36, a machine learning model for data (hereinafter abbreviated as the data model) 37, and an expression level set 38.

[0032] When the operating program 30 is started, the CPU 22 of the computer constituting the cell culture evaluation device 10 works in cooperation with the memory 21 and the like to function as a Read Write (hereinafter abbreviated as RW) control unit 45, a first processing unit 46, an aggregation unit 47, a second processing unit 48, and a display control unit 49.

[0033] The RW control unit 45 controls the storage of various data to the storage device 20 and the reading of various data from the storage device 20. For example, the RW control unit 45 receives cell images 12 from the imaging device 11 and stores them in the storage device 20. As a result, a group of cell images 35, consisting of multiple cell images 12 taken of each region 18 of one well 14, is stored in the storage device 20. Note that although only one group of cell images 35 is shown in Figure 4, multiple groups may be stored.

[0034] The RW control unit 45 reads a set of cell images 35 designated for predicting the expression levels of multiple types of ribonucleic acid X (see Figure 7) in cells 13 from the storage device 20 and outputs it to the first processing unit 46. As described above, the set of cell images 35 consists of multiple cell images 12. Therefore, the RW control unit 45 acquires the cell images 12 by reading the set of cell images 35 from the storage device 20. In other words, the RW control unit 45 is an example of an "acquisition unit" related to the technology of this disclosure.

[0035] Furthermore, the RW control unit 45 reads the image model 36 from the storage device 20 and outputs it to the first processing unit 46. In addition, the RW control unit 45 reads the data model 37 from the storage device 20 and outputs it to the second processing unit 48. The data model 37 predicts, for example, the expression level of ribonucleic acid X on the last day of culture. The data model 37 is a machine learning model such as a support vector machine, random forest, or neural network, and handles only one image feature set 55, which will be described later.

[0036] The first processing unit 46 inputs the cell image 12 to the image model 36. The image model 36 then outputs an image feature set 55 consisting of multiple types of image features Z (see Figure 5) related to the cell image 12. The first processing unit 46 outputs an image feature set 55 for each of the multiple cell images 12 that make up the cell image group 35. The first processing unit 46 outputs the image feature set group 56, which consists of multiple image feature sets 55 corresponding to the multiple cell images 12, to the aggregation unit 47.

[0037] The aggregation unit 47 aggregates multiple image feature sets 55 that constitute the image feature set group 56 into a single representative image feature set 55R that can be handled by the data model 37. The aggregation unit 47 outputs the representative image feature set 55R to the second processing unit 48. The representative image feature set 55R is an example of an "aggregated image feature set" related to the technology of this disclosure.

[0038] The second processing unit 48 inputs the representative image feature set 55R to the data model 37. Then, it outputs an expression level set 38, which consists of the expression levels of multiple types of ribonucleic acid X, from the data model 37. The second processing unit 48 outputs the expression level set 38 to the RW control unit 45. In Figure 4, only one expression level set 38 is drawn, similar to the cell image group 35, but multiple sets may be stored.

[0039] The RW control unit 45 stores the expression amount set 38 in the storage device 20. The RW control unit 45 also reads the expression amount set 38 from the storage device 20 and outputs it to the display control unit 49.

[0040] The display control unit 49 controls the display of various screens on the display 24. These screens include a selection screen 65 (see Figure 8) for specifying a group of cell images 35 for predicting expression levels, and a results display screen 70 (see Figure 9) for displaying an expression level set 38 as the predicted expression level.

[0041] As shown in Figure 5, the first processing unit 46 inputs each of the multiple cell images 12_1, 12_2, ..., 12_M that constitute the cell image group 35 into the image model 36. The image model 36 then outputs image feature sets 55_1, 55_2, ..., 55_M for each of the multiple cell images 12_1, 12_2, ..., 12_M. Hereinafter, M is the number of cell images 12, which may be several thousand.

[0042] Image feature set 55_1 consists of multiple types of image features Z1_1, Z2_1, ..., ZN_1. Similarly, image feature set 55_2 consists of multiple types of image features Z1_2, Z2_2, ..., ZN_2, and image feature set 55_M consists of multiple types of image features Z1_M, Z2_M, ..., ZN_M. Note that N is the number of image features, for example, several thousand.

[0043] As shown in Figure 6, the aggregation unit 47 calculates the average values ​​Z1AVE, Z2AVE, ..., and ZNAVE of each image feature Z1, Z2, ..., ZN that make up each image feature set 55_1, 55_2, ..., 55_M. For example, the average value Z1AVE of image feature Z1 is Z1AVE = {Σ(Z1_K)} / M (where K = 1, 2, 3, ..., M). The aggregation unit 47 outputs the average values ​​Z1AVE, Z2AVE, ..., ZNAVE obtained in this way as the representative image feature set 55R.

[0044] As shown in Figure 7, the second processing unit 48 outputs the expression levels for each ribonucleic acid X1, X2, ..., XQ as an expression level set 38, as shown in Table 60. X1, X2, ..., XQ include markers such as "Nanog", "Oct4", "Sox2", and "Tcf3". In addition, X1, X2, ..., XQ also include ribonucleic acids other than markers whose causal relationship between expression level and cell quality 13 has not yet been elucidated. The ribonucleic acid X whose expression level is to be predicted is set in advance by the culturing operator. Note that Q is the number of ribonucleic acid X whose expression level is to be predicted, for example, 100.

[0045] Figure 8 shows a selection screen 65 for specifying a group of cell images 35 to predict expression levels. The selection screen 65 includes an input box 66 for directly entering the folder of the group of cell images 35 to predict expression levels, and a reference button 67 for entering the folder of the group of cell images 35 to predict expression levels from Explorer. The operator enters the folder of the group of cell images 35 to predict expression levels and then selects the OK button 68. As a result, the corresponding group of cell images 35 is read from the storage device 20 by the RW control unit 45 and output to the first processing unit 46.

[0046] Figure 9 shows the results display screen 70, which displays the expression level set 38 as the predicted expression level. The results display screen 70 displays a table 60 and a bar graph 71 showing each ribonucleic acid X and its expression level. The results display screen 70 disappears when the confirmation button 72 is selected.

[0047] As shown in Figure 10, the image model 36 uses the compression unit 76 of an autoencoder (AE) 75. The AE 75 has a compression unit 76 and a restoration unit 77. The cell image 12 is input to the compression unit 76. The compression unit 76 converts the cell image 12 into an image feature set 55. The compression unit 76 passes the image feature set 55 to the restoration unit 77. The restoration unit 77 generates a restored image 78 of the cell image 12 from the image feature set 55.

[0048] As shown in Figure 11, in the learning phase before the compression unit 76 is repurposed as the image model 36, the AE75 is input with a training cell image 12L and learns from it. The AE75 outputs a training reconstructed image 78L based on the training cell image 12L. Based on these training cell images 12L and training reconstructed images 78L, the AE75 performs a loss calculation using a loss function such as the Mean Squared Error (MSE). Then, according to the result of the loss calculation, the various coefficients of the AE75 are updated, and the AE75 is updated according to the update settings.

[0049] During the AE75's learning phase, the above series of processes—inputting the training cell image 12L into the AE75, outputting the training reconstructed image 78L from the AE75, loss calculation, update settings, and updating the AE75—are repeated while the training cell image 12L is exchanged. The repetition of the above series of processes ends when the reconstruction accuracy from the training cell image 12L to the training reconstructed image 78L reaches a predetermined set level. The compression unit 76 of the AE75, whose reconstruction accuracy has reached the set level, is stored in the storage device 20 and used as the image model 36.

[0050] As shown in Figure 12, the compression unit 76 has four extraction units 85_1, 85_2, 85_3, and 85_4 and a full-connection unit 86. The extraction units 85_1, 85_2, 85_3, and 85_4 extract target group feature maps 87_1, 87_2, 87_3, and 87_4, respectively. The extraction units 85_1, 85_2, 85_3, and 85_4 output the target group feature maps 87_1, 87_2, 87_3, and 87_4 to the full-connection unit 86. The full-connection unit 86 converts the target group feature maps 87_1, 87_2, 87_3, and 87_4 into an image feature set 55. The target group feature maps 87_1, 87_2, 87_3, and 87_4 are examples of "target group feature sets" related to the technology of this disclosure. In the following, if there is no need to distinguish between extraction units 85_1, 85_2, 85_3, and 85_4, they may be collectively referred to as extraction unit 85. Similarly, the target group feature maps 87_1, 87_2, 87_3, and 87_4 may be collectively referred to as target group feature map 87.

[0051] The target group feature map 87_1 consists of multiple types of target group features C1, C2, ..., CG. Similarly, the target group feature map 87_2 consists of multiple types of target group features D1, D2, ..., DH, and the target group feature map 87_3 consists of multiple types of target group features E1, E2, ..., EI. The target group feature map 87_4 consists of multiple types of target group features F1, F2, ..., FJ. Note that G, H, I, and J are the number of target group features C, D, E, and F, respectively, and are, for example, tens of thousands to hundreds of thousands.

[0052] As shown in Figure 13, the extraction unit 85_1 is for the first group of subjects to be extracted from the cell image 12. Similarly, the extraction unit 85_2 is for the second group of subjects to be extracted, the extraction unit 85_3 is for the third group of subjects to be extracted, and the extraction unit 85_4 is for the fourth group of subjects to be extracted.

[0053] The first extraction target group, handled by the extraction unit 85_1, is the smallest of all the extraction target groups. Therefore, the target group feature map 87_1 extracted by the extraction unit 85_1 represents the features of the relatively small extraction target groups within the cell image 12. In contrast, the fourth extraction target group, handled by the extraction unit 85_4, is the largest of all the extraction target groups. Therefore, the target group feature map 87_4 extracted by the extraction unit 85_4 represents the features of the relatively large extraction target groups within the cell image 12.

[0054] The second target group to be extracted, handled by the extraction unit 85_2, and the third target group to be extracted, handled by the extraction unit 85_3, are medium-sized target groups between the first and fourth target groups. Therefore, the target group feature maps 87_2 and 87_3 extracted by the extraction units 85_2 and 85_3 represent the characteristics of medium-sized target groups, which are between very small target groups and very large target groups.

[0055] As shown in Figures 14 and 15, the extraction unit 85 has two convolutional layers 90 and 91. In addition, the extraction units 85 other than extraction unit 85_4 have a pooling layer 92. The fully connected unit 86 has a fully connected layer 95. The convolutional layers 90 and 91 are represented as "conv" (abbreviation for convolution), and the pooling layer 92 is represented as "pool" (abbreviation for pooling). The fully connected layer 95 is represented as "fc" (abbreviation for fully connected). Although not shown in the figures, the extraction unit 85 incorporates an activation function and batch normalization.

[0056] The convolutional layers 90 and 91 perform a convolution process, for example, as shown in Figure 16. That is, a 3x3 filter 102 is applied to the input data 101, which has multiple pixels 100 arranged in two dimensions. Then, the pixel value e of one of the pixels 100, the pixel of interest 100I, is convolved with the pixel values ​​a, b, c, d, f, g, h, i of eight pixels 100S adjacent to the pixel of interest 100I. This yields output data 105, which, like the input data 101, has multiple pixels 104 arranged in two dimensions. The pixel values ​​of pixels 104 in the output data 105 are the target group features C, D, E, and F.

[0057] If the coefficients of filter 102 are r, s, t, u, v, w, x, y, and z, the pixel value k of the pixel 104I of interest in the output data 105, which is the result of the convolution operation on the pixel 100I of interest, can be obtained, for example, by calculating equation (1) below. k=az+by+cx+dw+ev+fu+gt+hs+ir...(1)

[0058] In the convolution process, the above convolution operation is sequentially performed on each pixel 100 of the input data 101, and the pixel value of pixel 104 of the output data 105 is output. In this way, the output data 105 is output, which has the pixel value of pixel 100 of the input data 101 convolved into it.

[0059] Output data 105 is output once for each filter 102. If multiple types of filters 102 are applied to a single input data 101, output data 105 is output for each filter 102. In other words, as shown in Figure 17, there are as many output data 105 as there are filters 102 applied to the input data 101. Also, since output data 105 has multiple pixels 104 arranged in two dimensions, it has a width and a height. When multiple output data 105 are output, the target group feature map 87 becomes a collection of multiple output data 105. In the target group feature map 87, the number of output data 105 is called the number of channels. Figure 17 illustrates a 4-channel target group feature map 87, which has 4 output data 105 output by applying 4 filters 102 to the input data 101.

[0060] Figure 18 is an explanatory diagram illustrating how the convolution process using filter 102 is applied to the concept of a convolutional neural network. First, assume that the convolutional neural network has an input layer IL and an output layer OL, each having multiple units U. In this case, the weights W1, W2, W3, ... which indicate the strength of the connections between each unit U in the input layer IL and each unit U in the output layer OL, correspond to the coefficients r~z of filter 102. Each unit U in the input layer IL is input with the pixel values ​​a, b, c, ... of each pixel 100 in the input data 101. The sum of the products of each pixel value a, b, c, ... and the weights W1, W2, W3, ... becomes the output value of each unit U in the output layer OL. This output value corresponds to the pixel value k of the output data 105. In the learning phase of AE75 shown in Figure 11, the coefficients r~z of filter 102, which correspond to the weights W1, W2, W3, ... are updated.

[0061] The pooling layer 92 performs pooling on the target group feature map 87. The pooling process involves calculating local statistics for the target group feature map 87 and reducing its size (width × height) to create a reduced target group feature map 87S.

[0062] As a pooling process, for example, the maximum value pooling process shown in Figure 19 is performed. In Figure 19, the maximum value pooling process is a process in which, for example, the maximum value of a pixel within a 2x2 pixel block 110 of the target group feature map 87 is obtained as a local statistic, and the obtained maximum value is used as the pixel value of the pixel in the reduced target group feature map 87S. If the maximum value pooling process is performed while shifting the block 110 by one pixel in the width and height directions, the reduced target group feature map 87S will be reduced to half the size of the original target group feature map 87.

[0063] Figure 19 illustrates the case where, among pixel values ​​a, b, e, and f within block 110A, b within pixel values ​​b, c, f, and g within block 110B, and h within pixel values ​​c, d, g, and h within block 110C are all at their maximum values. Alternatively, mean pooling can be used to obtain the mean value as the local statistic instead of the maximum value.

[0064] In Figures 14 and 15, the numbers such as 32 and 64 written on each target group feature map 87 indicate the number of output data 105, i.e., the number of channels, that each target group feature map 87 has. The numbers in parentheses such as 1 / 1 and 1 / 2 written in the lower left of each extraction unit 85_1, 85_2, 85_3, and 85_4 respectively indicate the size of the input data 101 handled by each extraction unit 85_1, 85_2, 85_3, and 85_4, based on the size of the cell image 12.

[0065] In Figure 14, the cell image 12 is input as input data 101 to the extraction unit 85_1, and two convolutional processes are performed on the cell image 12 by two convolutional layers 90_1 and 91_1. First, the cell image 12 is subjected to a convolutional process by the convolutional layer 90_1, which applies 32 filters 102 to the cell image 12, and a 32-channel intermediate target group feature map 87_1M is extracted. Then, the intermediate target group feature map 87_1M is subjected to a convolutional process by the convolutional layer 91_1, which applies another 32 filters 102 to the intermediate target group feature map 87_1M, and finally a 32-channel target group feature map 87_1 is extracted. The target group feature map 87_1 is output to the fully connected layer 95 of the fully connected unit 86.

[0066] The size of the target group feature map 87_1 finally extracted in the extraction unit 85_1 is the same as the size of the cell image 12. Therefore, the size handled in the extraction unit 85_1 is the same as the cell image 12, i.e., 1 / 1, representing the same size.

[0067] The pooling layer 92_1 performs maximum pooling on the target group feature map 87_1, reducing the target group feature map 87_1 to half its size, resulting in a reduced target group feature map 87_1S. The pooling layer 92_1 outputs the reduced target group feature map 87_1S to the extraction unit 85_2. In other words, the extraction unit 85_2 receives the reduced target group feature map 87_1S, which has been reduced to half its size based on the size of the cell image 12, as input data 101.

[0068] In the extraction unit 85_2, the reduced target group feature map 87_1S from the extraction unit 85_1 is subjected to two convolution processes by convolutional layers 90_2 and 91_2, applying 64 filters 102 to it, ultimately extracting a 64-channel target group feature map 87_2. The target group feature map 87_2 is output to the fully connected layer 95 of the fully connected unit 86.

[0069] The pooling layer 92_2 performs maximum pooling on the target group feature map 87_2, reducing the target group feature map 87_2 to half its size, resulting in a reduced target group feature map 87_2S (see Figure 15). The pooling layer 92_2 outputs the reduced target group feature map 87_2S to the extraction unit 85_3. In other words, the extraction unit 85_3 receives the reduced target group feature map 87_2S, which has been reduced to one-quarter of the size of the cell image 12, as input data 101.

[0070] In Figure 15, the extraction unit 85_3 performs two convolution operations on the reduced target group feature map 87_2S from the extraction unit 85_2, applying 128 filters 102 by convolutional layers 90_3 and 91_3, ultimately extracting a 128-channel target group feature map 87_3. The target group feature map 87_3 is output to the fully connected layer 95 of the fully connected unit 86.

[0071] The pooling layer 92_3 performs maximum pooling on the target group feature map 87_3, reducing the target group feature map 87_3 to half its size, resulting in a reduced target group feature map 87_3S. The pooling layer 92_3 outputs the reduced target group feature map 87_3S to the extraction unit 85_4. In other words, the extraction unit 85_4 receives the reduced target group feature map 87_3S, which has been reduced to 1 / 8 of the size of the cell image 12, as input data 101.

[0072] In the extraction unit 85_4, the convolution process is performed twice on the reduced target group feature map 87_3S from the extraction unit 85_3, applying 256 filters 102 by the convolutional layers 90_4 and 91_4, ultimately extracting a 256-channel target group feature map 87_4. The target group feature map 87_4 is output to the fully connected layer 95 of the fully connected unit 86.

[0073] In this way, the input data 101 (cell image 12 or reduction target group feature map 87S) input to each extraction unit 85_1, 85_2, 85_3, and 85_4 is gradually reduced in size and resolution from the top extraction unit 85_1 to the bottom extraction unit 85_4. In this example, based on the size of the cell image 12, the input data 101 is input to extraction unit 85_1 at 1 / 1 (1x), to extraction unit 85_2 at 1 / 2, to extraction unit 85_3 at 1 / 4, and to extraction unit 85_4 at 1 / 8. The reason for increasing the number of filters 102 from extraction unit 85_1 to extraction unit 85_4 (32, 64, ...) is to increase the number of filters 102 as the size of the input data 101 handled decreases, in order to extract various features contained in the cell image 12.

[0074] As shown in Figure 20, the fully connected layer 95 has an input layer IL with units U equal to the number of target group features C, D, E, and F, and an output layer OL with units U equal to the number of image features Z. Each unit U in the input layer IL and each unit U in the output layer OL are fully connected to each other, and each has a weight assigned to it. Each unit U in the input layer IL is input with the target group features C, D, E, and F. The sum of the products of the target group features C, D, E, and F and the weights assigned between each unit U becomes the output value of each unit U in the output layer OL. This output value is the image feature Z.

[0075] Although not shown in the diagram, the reconstruction unit 77 also has a fully connected section. Unlike the fully connected section 86 of the compression unit 76, the fully connected section of the reconstruction unit 77 converts the image features Z of the image feature set 55 from the compression unit 76 into target group features corresponding to the target group features F. The reconstruction unit 77 then gradually expands the resulting target group feature map 87, in the opposite direction to the compression unit 76, ultimately obtaining the reconstructed image 78. In this process of gradually expanding the target group feature map 87, the reconstruction unit 77 performs convolution processing using convolutional layers. This type of processing is called upconvolution.

[0076] Figure 21 shows an overview of the processing during the training phase of the data model 37. During the training phase, the data model 37 is trained using training data 115. The training data 115 consists of a training image feature set 55L and a set of ground truth expression levels 38CA corresponding to the training image feature set 55L. The training image feature set 55L is obtained by inputting a cell image 12 into the image model 36. The ground truth expression level set 38CA is the result of actually measuring the expression levels of multiple types of ribonucleic acid X in cells 13 that appear in the cell image 12 when the training image feature set 55L was obtained. Methods for measuring expression levels include Q-PCR (Quantitative polymerase chain reaction), RNA sequencing, and single-cell RNA sequencing.

[0077] During the learning phase, the data model 37 is input with the training image feature set 55L. The data model 37 outputs the training expression set 38L based on the training image feature set 55L. Based on this training expression set 38L and the ground truth expression set 38CA, the data model 37 performs a loss calculation. Then, based on the result of the loss calculation, the various coefficients of the data model 37 are updated, and the data model 37 is updated according to the update settings.

[0078] During the training phase of the data model 37, the above series of processes—input of the training image feature set 55L into the data model 37, output of the training expression set 38L from the data model 37, loss calculation, update settings, and updating the data model 37—are repeated while the training data 115 is exchanged. The repetition of the above series of processes ends when the prediction accuracy of the training expression set 38L for the ground truth expression set 38CA reaches a predetermined set level. The data model 37, whose prediction accuracy has thus reached the set level, is stored in the storage device 20 and used by the second processing unit 48.

[0079] Next, the operation of the above configuration will be explained with reference to the flowchart in Figure 22. First, when the operating program 30 is started in the cell culture evaluation device 10, the CPU 22 of the cell culture evaluation device 10 functions as the RW control unit 45, the first processing unit 46, the aggregation unit 47, the second processing unit 48, and the display control unit 49, as shown in Figure 4.

[0080] In the selection screen 65 shown in Figure 8, if a group of cell images 35 for predicting expression levels is selected (YES in step ST100), the RW control unit 45 reads the corresponding group of cell images 35 and image model 36 from the storage device 20 (step ST110). The group of cell images 35 and image model 36 are output from the RW control unit 45 to the first processing unit 46. Step ST110 is an example of "acquisition" related to the technology of this disclosure.

[0081] In the first processing unit 46, as shown in Figure 5, one of the multiple cell images 12 constituting the cell image group 35 is input to the image model 36. Then, an image feature set 55 is output from the image model 36 (step ST120). The processing in step ST120 continues until an image feature set 55 is output for all of the multiple cell images 12 constituting the cell image group 35 (NO in step ST130). Note that step ST120 is an example of the "first processing" related to the technology of this disclosure.

[0082] If an image feature set 55 is output for all of the multiple cell images 12 that make up the cell image group 35 (YES in step ST130), an image feature set group 56 consisting of multiple image feature sets 55 is generated. The image feature set group 56 is output from the first processing unit 46 to the aggregation unit 47.

[0083] As shown in Figure 6, in the aggregation unit 47, multiple image feature sets 55 that constitute the image feature set group 56 are aggregated into one representative image feature set 55R that can be handled by the data model 37 (step ST140). The representative image feature set 55R is output from the aggregation unit 47 to the second processing unit 48.

[0084] The second processing unit 48 receives the data model 37 read from the storage device 20 by the RW control unit 45. In the second processing unit 48, as shown in Figure 7, the representative image feature set 55R is input to the data model 37. Then, the expression amount set 38 is output from the data model 37 (step ST150). The expression amount set 38 is output from the second processing unit 48 to the RW control unit 45. Step ST150 is an example of the "second processing" related to the technology of this disclosure.

[0085] The expression amount set 38 is stored in the storage device 20 by the RW control unit 45. The expression amount set 38 is also read from the storage device 20 by the RW control unit 45 and output to the display control unit 49.

[0086] Under the control of the display control unit 49, the result display screen 70 shown in Figure 9 is displayed on the display 24 (step ST160). The operator confirms the expression level set 38 through the result display screen 70.

[0087] As described above, the CPU 22 of the cell culture evaluation device 10 functions as an acquisition unit: an RW control unit 45, a first processing unit 46, and a second processing unit 48. The RW control unit 45 acquires cell images 12 by reading a group of cell images 35 from the storage device 20. The first processing unit 46 inputs the cell images 12 into an image model 36 and outputs an image feature set 55 from the image model 36, which consists of multiple types of image features Z related to the cell images 12. The second processing unit 48 inputs the image feature set 55 (representative image feature set 55R) into a data model 37 and outputs an expression level set 38 from the data model 37, which consists of the expression levels of multiple types of ribonucleic acid X of cells 13.

[0088] The image feature quantity Z was not obtained by inputting the cell image 12 into commercially available image analysis software, as described in Japanese Patent Publication No. 2009-044974, but rather by inputting the cell image 12 into an image model 36. Therefore, the image feature quantity Z is not something that can be intuitively perceived by humans visually, like the index values ​​described in Japanese Patent Publication No. 2009-044974, nor is it arbitrarily set by humans. The image feature quantity Z does not represent limited characteristics of the cell 13, like the index values ​​described in Japanese Patent Publication No. 2009-044974, but rather represents comprehensive characteristics of the cell 13. Therefore, it is possible to appropriately predict the expression levels of multiple types of ribonucleic acid X. As a result, it is possible to further accelerate the elucidation of the causal relationship between expression levels and the quality of the cell 13, and greatly contribute to improving the quality of the cell 13. In addition, compared to methods that actually measure expression levels, such as Q-PCR, it is possible to obtain expression level data in a non-invasive and low-cost manner.

[0089] The display control unit 49 controls the display of the expression level set 38. This ensures that the operator is reliably informed of the predicted expression level.

[0090] The RW control unit 45 reads the cell image group 35 from the storage device 20 to acquire multiple cell images 12 obtained by taking multiple images of a single well 14 in which cells 13 are cultured. The first processing unit 46 inputs each of the multiple cell images 12 into the image model 36 and causes the image model 36 to output an image feature set 55 for each of the multiple cell images 12. As a result, an expression level set 38 can be output based on the image feature set 55 output for each of the multiple cell images 12, thereby increasing the reliability of the expression level set 38.

[0091] The aggregation unit 47 aggregates the multiple image feature sets 55 output for each of the multiple cell images 12 into a single representative image feature set 55R that can be handled by the data model 37. The second processing unit 48 inputs the representative image feature set 55R into the data model 37 and causes the data model 37 to output an expression level set 38. This ensures that the expression level set 38 can be reliably output from the data model 37.

[0092] As shown in Figure 10, the compression unit 76 of the AE75, which has a compression unit 76 that converts the cell image 12 into an image feature set 55 and a restoration unit 77 that generates a reconstructed image 78 of the cell image 12 from the image feature set 55, is used as the image model 36. Therefore, as shown in Figure 11, the only training data that needs to be prepared in the training phase is the training cell image 12L. Thus, the image model 36 can be obtained very easily without incurring extra costs and time.

[0093] As shown in Figures 12 to 15, the compression unit 76 has multiple extraction units 85 and a fully connected unit 86. Multiple extraction units 85 are provided according to the size of the target groups to be extracted in the cell image 12. Each extraction unit 85 uses convolutional layers 90 and 91 to extract a target group feature map 87 consisting of multiple types of target group feature quantities C, D, E, and F for the target groups it is responsible for. The fully connected unit 86 uses a fully connected layer 95 to convert the multiple target group feature maps 87 output from the multiple extraction units 85 into an image feature set 55. As a result, a wide range of target group feature quantities C, D, E, and F can be obtained, from target group feature maps 87 representing the features of relatively fine target groups in the cell image 12 to target group feature maps 87 representing the features of relatively large target groups in the cell image 12. This, in turn, can improve the comprehensiveness of the image feature quantities Z.

[0094] Note that while Figure 9 shows an example of displaying the expression level itself, this is not the only option. Normalized expression levels, where the expression level of a baseline ribonucleic acid X is set to 1, may also be displayed. Furthermore, a pie chart may be displayed instead of, or in addition to, the bar graph 71.

[0095] [Second Embodiment] In the second embodiment shown in Figures 23 to 25, a Generative Adversarial Network (GAN) 120 is used in the learning phase of AE75.

[0096] As shown in Figure 23, in the second embodiment, GAN120 is used in the loss calculation of AE75 using a loss function such as mean squared error. Specifically, as shown in Figure 24, AE75 is incorporated as a generator for GAN120. The discriminator 121 then determines whether the training cell image 12L input to AE75 and the training reconstructed image 78L output from AE75 are identical. The discriminator 121 outputs the determination result 122.

[0097] As shown in Figure 25A, if the determination result 122A indicates that the training cell image 12L and the training reconstructed image 78L are not identical, the AE75 loss is estimated to be large. Therefore, in this case, the AE75 update amount is set to be relatively large. Conversely, as shown in Figure 25B, if the determination result 122B indicates that the training cell image 12L and the training reconstructed image 78L are identical, the AE75 loss is estimated to be small. Therefore, in this case, the AE75 update amount is set to be relatively small.

[0098] Thus, in the second embodiment, AE75 is trained using a GAN120 that has a discriminator 121 that determines whether the training cell image 12L and the training reconstructed image 78L are identical. If only a loss function such as mean squared error is used, the reconstruction accuracy from the training cell image 12L to the training reconstructed image 78L will plateau at a certain level. In contrast, by using GAN120, it is possible to further improve the reconstruction accuracy beyond the level that plateaued when only a loss function such as mean squared error was used. As a result, the reliability of the image feature Z can be increased, and consequently, the reliability of the expression set 38 can also be increased.

[0099] [Third Embodiment] In the third embodiment shown in Figures 26 and 27, in addition to the image feature set 55 from the compression unit 76, morphological information 130 of the cells 13 is input to the reconstruction unit 77 to train the AE75. That is, as shown in Figure 26, in the third embodiment, during the training phase of the AE75, morphological information 130 of the cells 13 depicted in the training cell image 12L is input to the reconstruction unit 77 along with the image feature set 55 from the compression unit 76. The reconstruction unit 77 uses the morphological information 130 as a reference to reconstruct the image feature set 55 into the training reconstructed image 78L.

[0100] As shown in Figure 27, the morphology-related information 130 includes items such as the type of cell 13, the donor, confluence, quality, and reprogramming method. The type item registers cell types such as the example cardiomyocytes, nerve cells, and hepatocytes. Alternatively, the type item may register hematopoietic stem cells, mesenchymal stem cells, skin stem cells, progenitor cells, etc. The donor item registers the donor's race, sex, and age. The confluence item registers the percentage of the area where cells 13 occupy the entire area of ​​well 14. The quality item registers the quality level of the cells 13 as determined by a well-known judgment method. Specifically, in addition to the example good, average and poor are registered. The reprogramming method item registers the method used to reprogram (also called) the cells 13, such as the example RNA introduction method. In practice, numbers indicating the type of cell 13, donor, quality, and reprogramming method are registered. In addition to these items, morphological information 130 also includes items such as the number of days of culture.

[0101] Thus, in the third embodiment, the AE75 is trained by inputting morphological information 130 of the cells 13 into the reconstruction unit 77, in addition to the image feature set 55 from the compression unit 76. As a result, it becomes easier for the reconstruction unit 77 to reconstruct the image feature set 55 into a training image 78L, and the training of the AE75 can be completed in a short time.

[0102] The morphological information 130 includes the cell type, provider, confluence, quality, and initialization method of the cell 13. These are all important items that determine the morphology of the cell 13. Therefore, it becomes easier to reconstruct the training image 78L from the image feature set 55 in the reconstruction unit 77. Note that the morphological information 130 only needs to include at least one of the above-mentioned cell type, provider, confluence, quality, and initialization method.

[0103] [Fourth Embodiment] In the fourth embodiment shown in Figures 28 to 30, in addition to the image feature set 55, reference information 141 that serves as a reference for the output of the expression level set 38 is input to the data model 140.

[0104] In Figure 28, the second processing unit 48 inputs the image feature set 55 and reference information 141 to the data model 140. Then, it outputs the expression level set 38 from the data model 140. The reference information 141 includes morphological information 142 and culture supernatant component information 143. The morphological information 142 is information about the cells 13 shown in the cell image 12 from which the image feature set 55 was extracted. Similarly, the culture supernatant component information 143 is information about the cells 13 shown in the cell image 12 from which the image feature set 55 was extracted. The culture supernatant component information 143 is information about the culture supernatant collected from the well 14. The culture supernatant component information 143 is obtained by analyzing the culture supernatant with an analyzer.

[0105] As shown in Figure 29, the culture supernatant component information 143 includes items such as pH (Potential of Hydrogen), glucose, ATP (Adenosine Triphosphate), and lactic acid. The amounts of glucose, ATP, and lactic acid are registered in each item. Note that the morphological information 142 is the same as the morphological information 130 in the third embodiment described above, so its illustration and description are omitted.

[0106] Figure 30 shows an overview of the processing during the training phase of the data model 140. During the training phase, the data model 140 is trained using training data 145. The training data 145 includes the training image feature set 55L and the ground truth expression set 38CA shown in Figure 21 of the first embodiment, as well as training reference information 141L. The training reference information 141L includes training morphological information 142L and training culture supernatant component information 143L. Both the training morphological information 142L and the training culture supernatant component information 143L are information about cells 13 that appear in the cell image 12 from which the training image feature set 55L was extracted.

[0107] The data model 140 receives the training image feature set 55L and the training reference information 141L as input. The data model 140 outputs the training expression set 38L based on the training image feature set 55L and the training reference information 141L. The subsequent loss calculation and update setting processes are the same as in the first embodiment described above, so their explanation is omitted.

[0108] During the training phase of the data model 140, the above series of processes—inputting the training image feature set 55L and training reference information 141L into the data model 140, outputting the training expression set 38L from the data model 140, loss calculation, update settings, and updating the data model 140—are repeated while the training data 145 is exchanged. The repetition of the above series of processes ends when the prediction accuracy of the training expression set 38L for the ground truth expression set 38CA reaches a predetermined set level. The data model 140, whose prediction accuracy has thus reached the set level, is stored in the storage device 20 and used by the second processing unit 48.

[0109] Thus, in the fourth embodiment, the second processing unit 48 inputs reference information 141, which serves as a reference for the output of the expression level set 38, to the data model 140, in addition to the image feature set 55. This improves the prediction accuracy of the expression level set 38.

[0110] Reference information 141 includes morphological information 142 of cell 13 and information 143 of the culture supernatant components of cell 13. Morphological information 142 and culture supernatant component information 143 are useful for predicting the expression level set 38. Therefore, the prediction accuracy of the expression level set 38 can be further improved.

[0111] Morphology-related information 142 includes the cell type, donor, confluence, quality, and reprogramming method of the cell 13. These are all important factors that determine the morphology of the cell 13. Therefore, the prediction accuracy of the expression level set 38 can be improved. Note that, like morphology-related information 130, morphology-related information 142 only needs to include at least one of the above-mentioned cell type, donor, confluence, quality, and reprogramming method.

[0112] [Fifth Embodiment] In the fifth embodiment shown in Figures 31 and 32, the compression unit 151 of a convolutional neural network (CNN) 150 is used as the image model 155 instead of the compression unit 76 of AE75.

[0113] As shown in Figure 31, the CNN 150 has a compression unit 151 and an output unit 152. The cell image 12 is input to the compression unit 151. The compression unit 151, like the compression unit 76, converts the cell image 12 into an image feature set 153. The compression unit 151 passes the image feature set 153 to the output unit 152. The output unit 152 outputs evaluation labels 154 about the cell 13 based on the image feature set 153. The evaluation labels 154 are, for example, the quality of the cell 13 or the type of cell 13. This compression unit 151 of the CNN 150 is used as an image model 155.

[0114] Figure 32 shows an overview of the processing during the training phase of CNN150. During the training phase, CNN150 is trained using training data 158. Training data 158 consists of a training cell image 12L and a corresponding ground truth evaluation label 154CA. The ground truth evaluation label 154CA is obtained by actually evaluating the cells 13 shown in the training cell image 12L.

[0115] During the training phase, the CNN150 receives training cell images 12L as input. The CNN150 outputs training evaluation labels 154L for the training cell images 12L. Based on these training evaluation labels 154L and the ground truth evaluation labels 154CA, the CNN150 performs a loss calculation. Then, based on the result of the loss calculation, the various coefficients of the CNN150 are updated, and the CNN150 is updated according to these update settings.

[0116] During the training phase of CNN150, the above series of processes—inputting the training cell image 12L into CNN150, outputting the training evaluation label 154L from CNN150, loss calculation, update settings, and updating CNN150—are repeated while the training data 158 is exchanged. The repetition of the above series of processes ends when the prediction accuracy of the training evaluation label 154L for the ground truth evaluation label 154CA reaches a predetermined set level. The compression unit 151 of CNN150, whose prediction accuracy has reached the set level, is stored in the storage device 20 as an image model 155 and used by the first processing unit 46.

[0117] Thus, in the fifth embodiment, the compression unit 151 of the CNN 150, which has a compression unit 151 that converts cell images 12 into an image feature set 153 and an output unit 152 that outputs evaluation labels 154 for cells 13 based on the image feature set 153, is used as the image model 155. Therefore, if there is a sufficient number of training data 158 consisting of training cell images 12L and correct evaluation labels 154CA, the image model 155 can be created using this training data 158.

[0118] [Sixth Embodiment] In the first embodiment described above, the multiple cell images 12 were exemplified as cell images 12 taken for each of the multiple regions 18 obtained by dividing the well 14, but the invention is not limited to this. As in the sixth embodiment shown in Figures 33 and 34, the multiple cell images 12 may be cell images 12 obtained by taking images using different imaging methods, or cell images 12 obtained by taking images of cells 13 stained with different dyes.

[0119] Figure 33 shows an example in which multiple cell images 12 are obtained by taking images using different imaging methods. Specifically, there are two types of cell images 12: cell image 12mA obtained with imaging device 11mA using imaging method A, and cell image 12mB obtained with imaging device 11mB using imaging method B. Imaging device 11mA is, for example, a bright-field microscope, and imaging device 11mB is, for example, a phase-contrast microscope. In this case, the CPU 22 is configured with a first processing unit 160A and an aggregation unit 161A for cell image 12mA, a first processing unit 160B and an aggregation unit 161B for cell image 12mB, and a second processing unit 162.

[0120] The first processing unit 160A receives a cell image group 35mA and an image model 165A as inputs. The cell image group 35mA consists of multiple cell images 12mA, each captured by the imaging device 11mA for each of the multiple regions 18. The first processing unit 160A inputs the cell images 12mA to the image model 165A and causes the image model 165A to output an image feature set 55mA. The first processing unit 160A outputs an image feature set 55mA for each of the multiple cell images 12mA and outputs a group of image feature sets 56mA, which consists of multiple image feature sets 55mA, to the aggregation unit 161A. The aggregation unit 161A aggregates the multiple image feature sets 55mA into a representative image feature set 55mRA and outputs the representative image feature set 55mRA to the second processing unit 162. Note that the processing of the first processing unit 160B and the aggregation unit 161B is basically the same as the processing of the first processing unit 160A and the aggregation unit 161B, with only "A" being changed to "B", so the explanation is omitted.

[0121] The second processing unit 162 receives the representative image feature sets 55mRA and 55mRB, as well as the data model 166. The second processing unit 162 inputs the representative image feature sets 55mRA and 55mRB into the data model 166 and outputs the expression level set 38 from the data model 166. The data model 166 is trained using a training image feature set (not shown) extracted from cell images 12mA captured by imaging device 11mA, and a training image feature set (not shown) extracted from cell images 12mB captured by imaging device 11mB.

[0122] If multiple cell images 12 are obtained by capturing them using different imaging methods, the prediction accuracy of the expression level set 38 can be further improved. This is because, for example, cell images 12 captured with a bright-field microscope do not show phase objects, which are colorless and transparent objects, but cell images 12 captured with a phase-contrast microscope do. In other words, different imaging methods have their strengths and weaknesses. Therefore, considering multiple imaging methods comprehensively will further improve the prediction accuracy of the expression level set 38.

[0123] While bright-field microscopes are given as examples for imaging device 11mA and phase-contrast microscopes for imaging device 11mB, the examples are not limited to these. Dark-field microscopes, confocal microscopes, differential interference microscopes, modulation contrast microscopes, etc., may also be used. Furthermore, there may be three or more different imaging methods, not just two. In addition, instead of using a separate image model 165 for each imaging method, a common image model 165 may be used for multiple imaging methods.

[0124] Figure 34 shows an example in which multiple cell images 12 are obtained by photographing cells 13 stained with different dyes. Specifically, this includes cell image 12dA obtained by photographing cells 13 stained with dye A, and cell image 12dB obtained by photographing cells 13 stained with dye B using the same imaging device 11 as for dye A. Dye A is, for example, hematoxylin and eosin, and dye B is, for example, crystal violet. In this case, the CPU 22 is configured with a first processing unit 170A and an aggregation unit 171A for cell image 12dA, a first processing unit 170B and an aggregation unit 171B for cell image 12dB, and a second processing unit 172.

[0125] The first processing unit 170A receives a group of cell images 35dA and an image model 175A as input. The group of cell images 35dA consists of multiple cell images 12dA obtained by photographing cells 13 stained with dye A in multiple regions 18. The first processing unit 170A inputs the cell images 12dA to the image model 175A and outputs an image feature set 55dA from the image model 175A. The first processing unit 170A outputs an image feature set 55dA for each of the multiple cell images 12dA and outputs a group of image feature sets 56dA, which consists of multiple image feature sets 55dA, to the aggregation unit 171A. The aggregation unit 171A aggregates the multiple image feature sets 55dA into a representative image feature set 55dRA and outputs the representative image feature set 55dRA to the second processing unit 172. Note that the processing of the first processing unit 170B and the aggregation unit 171B is basically the same as the processing of the first processing unit 170A and the aggregation unit 171B, with only "A" being changed to "B", so the explanation is omitted.

[0126] The second processing unit 172 receives the representative image feature sets 55dRA and 55dRB, as well as the data model 176. The second processing unit 172 inputs the representative image feature sets 55dRA and 55dRB into the data model 176 and outputs the expression level set 38 from the data model 176. The data model 176 is trained using a training image feature set (not shown) extracted from cell image 12dA obtained by photographing cells 13 stained with dye A, and a training image feature set (not shown) extracted from cell image 12dB obtained by photographing cells 13 stained with dye B.

[0127] If multiple cell images 12 are obtained by photographing cells 13 stained with different dyes, the prediction accuracy of the expression level set 38 can be further improved. This is because, as in the case of Figure 33, some dyes stain the cell nucleus, while others stain glycans, and so on; different dyes have their strengths and weaknesses. Therefore, considering multiple dyes comprehensively will further improve the prediction accuracy of the expression level set 38.

[0128] Note that the dyes are not limited to hematoxylin, eosin, and crystal violet. Methylene blue, neutral red, Nile blue, etc., may also be used. Furthermore, there may be three or more different dyes, not just two. In addition, instead of using a separate image model 175 for each dye, a common image model 175 may be used for multiple dyes.

[0129] For example, the four cell images 12 obtained by photographing cells 13 stained with dyes A and B using imaging devices 11A and 11B, respectively, may be used as the target, and the methods shown in Figure 33 and Figure 34 may be combined. In addition, the multiple cell images 12 may be taken in chronological order, such as on day 1 and day 2 of culture.

[0130] In each of the embodiments described above, multiple image feature sets 55 are aggregated into a single representative image feature set 55R, but this is not limited to this. It is sufficient to aggregate them to a number that can be handled by the data model 37; for example, 1000 image feature sets 55 may be aggregated into 10.

[0131] Instead of calculating the mean values ​​Z1AVE, Z2AVE, ..., ZNAVE for each image feature Z1, Z2, ..., ZN, multiple image feature sets 55 may be aggregated by performing principal component analysis on each image feature Z1, Z2, ..., ZN.

[0132] The aggregation unit 47 is optional. An expression level set 38 may be output for each of the multiple image feature sets 55 extracted from multiple cell images 12. In this case, for example, an expression level set 38 is output for multiple cell images 12 taken for each of the multiple regions 18 obtained by dividing the well 14. Therefore, conventionally, due to the relationship between measurement cost and measurement time, only one expression level set 38 was available for each well 14, but by using the technology of this disclosure, multiple expression level sets 38 can be obtained for each well 14. In other words, the resolution of the expression level set 38 for each well 14 can be increased.

[0133] The hardware configuration of the computer constituting the cell culture evaluation device 10 can be modified in various ways. The cell culture evaluation device 10 can also be configured with multiple computers separated as hardware, for the purpose of improving processing power and reliability. For example, the functions of the RW control unit 45 and the display control unit 49, and the functions of the first processing unit 46, the aggregation unit 47, and the second processing unit 48 can be distributed among two computers. In this case, the cell culture evaluation device 10 is configured with two computers.

[0134] Thus, the hardware configuration of the computer in the cell culture evaluation device 10 can be appropriately changed according to the required performance, such as processing power, safety, and reliability. Furthermore, not only the hardware, but also application programs such as the operating program 30 can, of course, be duplicated or distributed and stored on multiple storage devices to ensure safety and reliability.

[0135] In each of the above embodiments, for example, the following types of processors can be used as the hardware extraction target group for processing units that perform various processes, such as the RW control unit 45, the first processing unit 46, 160A, 160B, 170A, 170B, the aggregation unit 47, 161A, 161B, 171A, 171B, the second processing unit 48, 162, 172, and the display control unit 49. The types of processors include a CPU 22, which is a general-purpose processor that executes software (operation program 30) and functions as various processing units, as well as programmable logic devices (PLDs) such as FPGAs (Field Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and / or dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) which have a circuit configuration specifically designed to perform a particular process.

[0136] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, and / or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0137] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured using one or more of the above-mentioned processors as hardware extraction targets.

[0138] Furthermore, more specifically, the hardware-based extraction targets for these various processors can be electrical circuits (Circuitry) that combine circuit elements such as semiconductor devices.

[0139] The technology of this disclosure can be appropriately combined with the various embodiments and / or variations described above. Furthermore, it is understood that various configurations can be adopted without departing from the spirit of the invention, and the invention is not limited to the embodiments described above. Moreover, the technology of this disclosure extends not only to programs but also to storage media for storing programs non-temporarily.

[0140] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0141] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0142] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. Equipped with at least one processor, The aforementioned processor, Cell images obtained by photographing cells in culture are acquired. The cell image is input into an image machine learning model, and the image machine learning model outputs an image feature set consisting of multiple types of image features related to the cell image. The aforementioned image feature set is input into a machine learning model for data, and the machine learning model for data outputs an expression level set consisting of the expression levels of multiple types of ribonucleic acids in the aforementioned cells. It is structured in such a way, The aforementioned machine learning model for images is obtained by inputting a training cell image into the model, outputting a set of image features of the training cell image, generating a reconstructed training image from the set of image features, and updating the model according to the result of a loss calculation based on the training cell image and the reconstructed training image. The aforementioned machine learning model for the data is obtained by inputting a set of training image features into the model, outputting a set of training expression levels for the set of training image features, and updating the model according to the result of a loss calculation based on the set of training expression levels and the set of ground truth expression levels. Cell culture evaluation device.

2. The aforementioned processor, The cell culture evaluation apparatus according to claim 1, configured to perform control to display the expression level set.

3. The aforementioned processor, Multiple images of the cells are obtained by taking multiple photographs of a single culture vessel in which the cells are cultured. A cell culture evaluation apparatus according to claim 1 or claim 2, configured to input each of the multiple cell images into the image machine learning model, and to output the image feature set for each of the multiple cell images from the image machine learning model.

4. The aforementioned processor, The multiple image feature sets output for each of the multiple cell images are aggregated into a number that can be handled by the machine learning model for the data. The cell culture evaluation apparatus according to claim 3, configured to input an aggregated set of image features into the machine learning model for data, and to output the expression level set for each of the aggregated set of image features from the machine learning model for data.

5. The cell culture evaluation apparatus according to claim 3 or claim 4, wherein the multiple cell images include at least one of cell images obtained by taking images using different imaging methods and cell images obtained by taking images of cells stained with different dyes.

6. The aforementioned processor, The cell culture evaluation apparatus according to any one of claims 1 to 5, wherein, in addition to the image feature set, reference information that serves as a reference for the output of the expression level set is input to the machine learning model for the data.

7. The cell culture evaluation apparatus according to claim 6, wherein the reference information includes morphological information of the cells and information on the components of the culture supernatant of the cells.

8. The cell culture evaluation apparatus according to claim 7, wherein the morphological information includes at least one of the cell type, provider, confluence, quality, and initialization method.

9. A compression unit that converts the cell image into the image feature set, A restoration unit that generates a reconstructed image of the cell image from the aforementioned set of image features, A cell culture evaluation apparatus according to any one of claims 1 to 8, wherein only the compression unit of an autoencoder having is used as the machine learning model for the image.

10. The compression section is The extraction units are arranged according to the size of the target group to be extracted in the cell image, and each extraction unit uses a convolutional layer to extract a set of target group features consisting of multiple types of target group features for the target group it is responsible for, A fully connected layer is used to convert the multiple target group feature sets output from the multiple extraction units into the image feature set, A cell culture evaluation apparatus according to claim 9, having the following features.

11. The cell culture evaluation apparatus according to claim 9 or 10, wherein the autoencoder is trained using a generative adversarial network having a discriminator that determines whether the cell image and the reconstructed image are identical.

12. The cell culture evaluation apparatus according to any one of claims 9 to 11, wherein the autoencoder is learned by inputting the cell morphology-related information into the reconstruction unit in addition to the image feature set from the compression unit.

13. The cell culture evaluation apparatus according to claim 12, wherein the morphological information includes at least one of the cell type, provider, confluence, quality, and reprogramming method.

14. Cell images obtained by photographing cells in culture are acquired. The cell image is input into an image machine learning model, and a first process is performed to output an image feature set consisting of multiple types of image features related to the cell image from the image machine learning model. A second process is performed in which the aforementioned image feature set is input into a machine learning model for data, and the machine learning model for data outputs an expression level set consisting of the expression levels of multiple types of ribonucleic acids in the cells. This includes, The aforementioned machine learning model for images is obtained by inputting a training cell image into the model, outputting a set of image features of the training cell image, generating a reconstructed training image from the set of image features, and updating the model according to the result of a loss calculation based on the training cell image and the reconstructed training image. The aforementioned machine learning model for the data is obtained by inputting a set of training image features into the model, outputting a set of training expression levels for the set of training image features, and updating the model according to the result of a loss calculation based on the set of training expression levels and the set of ground truth expression levels. A method for operating a cell culture evaluation device performed by a processor.

15. An operating program for a cell culture evaluation device having a processor, An acquisition unit that acquires cell images obtained by photographing cells in culture, A first processing unit inputs the cell image into an image machine learning model and outputs an image feature set composed of multiple types of image features related to the cell image from the image machine learning model. A second processing unit inputs the aforementioned image feature set into a machine learning model for data, and outputs an expression level set consisting of the expression levels of multiple types of ribonucleic acids in the aforementioned cells from the machine learning model for data, To enable the aforementioned processor, The aforementioned machine learning model for images is obtained by inputting a training cell image into the model, outputting a set of image features of the training cell image, generating a reconstructed training image from the set of image features, and updating the model according to the result of a loss calculation based on the training cell image and the reconstructed training image. The aforementioned machine learning model for the data is obtained by inputting a set of training image features into the model, outputting a set of training expression levels for the set of training image features, and updating the model according to the result of a loss calculation based on the set of training expression levels and the set of ground truth expression levels. Operating program for cell culture evaluation device.

Images