Computer Program and Output Device

Abstract

To provide a computer program or the like that specifies a cell and determines the state of the cell without specifying an individual cell region.SOLUTION: A computer program causes a computer to perform the processes of: specifying a target region including a nucleus of a target cell, based on an image obtained by imaging a plurality of cells; extracting an image of the target region from each of multiple types of images including the image and an image corresponding to the image and of a type different from that of the image; and specifying and outputting the type of the target cell based on the extracted multiple images of the target region.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] The present invention relates to a computer program for determining the type of a cell. [Background technology]

[0002] In drug discovery, compounds are added at various concentrations to a sample containing cultured cells, and the state of the cells is judged at a predetermined timing. Conventionally, such cell state judgments have been made by skilled personnel visually observing the cells under a microscope. In recent years, in order to make stable judgments that are not influenced by the subjectivity of the judge, a technology has been proposed for detecting cells and judging their state by image analysis using images of the sample (Patent Document 1, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6660712 [Patent Document 2] JP 2018-180635 A Summary of the Invention [Problem to be solved by the invention]

[0004] To identify a cell, it is necessary to identify the area of ​​each individual cell, but since the cytoplasm that constitutes the outer periphery of the cell is highly transparent, it is difficult to distinguish the boundaries of each individual cell. The present invention has been made in consideration of this situation. The purpose of the present invention is to provide a computer program or the like that identifies a cell and determines the state of the cell without identifying the area of ​​each individual cell. [Means for solving the problem]

[0005] A computer program according to one embodiment of the present application causes a computer to execute a process of identifying a target area containing the nucleus of a target cell based on images of multiple cells, extracting images of the target area from each of multiple types of images including the image and an image that corresponds to the image but is of a different type from the image, and identifying and outputting the type of the target cell based on the multiple images of the extracted target area. Effect of the Invention

[0006] In one aspect of the present application, it becomes possible to identify cells and determine the state of the cells without identifying individual cell regions. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of a cell analysis system. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing device. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of a display terminal; FIG. [Figure 4] 13 is a flowchart illustrating an example of an analysis process procedure. [Diagram 5] FIG. 11 is an explanatory diagram showing a part of the analysis process. [Figure 6] FIG. 11 is an explanatory diagram showing a part of the analysis process. [Figure 7] FIG. 11 is an explanatory diagram showing a part of the analysis process. [Figure 8] FIG. 11 is an explanatory diagram showing an example of an input attribute DB. [Figure 9] FIG. 13 is an explanatory diagram showing an example of a result DB. [Figure 10] FIG. 13 is an explanatory diagram showing an example of a result details DB. [Figure 11] FIG. 2 is an explanatory diagram showing an example of a first training data DB. [Figure 12] FIG. 11 is an explanatory diagram showing an example of a second training data DB. [Figure 13] FIG. 1 is an explanatory diagram of the generation of a first learning model. [Figure 14]11 is a flowchart showing an example of a procedure for a first learning model generation process. [Figure 15] FIG. 11 is an explanatory diagram of mask image generation. [Figure 16] 13 is a flowchart illustrating an example of a procedure for mask image generation processing. [Figure 17] 13 is a flowchart showing an example of steps of a second learning model generation process. [Figure 18] 13 is a flowchart illustrating an example of a procedure for a determination process. [Figure 19] FIG. 13 is an explanatory diagram showing an example of a result display screen. [Figure 20] 13 is a flowchart showing an example of a procedure for a result display process. [Figure 21] FIG. 2 is a block diagram showing an example of functional units included in the image processing apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Embodiment 1) Hereinafter, an embodiment will be described with reference to the drawings. Fig. 1 is an explanatory diagram showing a configuration example of a cell analysis system. The cell analysis system 100 includes an image processing device 1 (output device), an imaging device 2, and a display terminal 3. The imaging device 2 is connected to the display terminal 3. The image processing device 1 and the display terminal 3 are connected to each other via a network N so as to be able to communicate with each other.

[0009] The image processing device 1 is configured with a server computer, a PC (Personal Computer), etc. The image processing device 1 may also be configured with a multicomputer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. Furthermore, the functions of the image processing device 1 may be realized by a cloud service.

[0010] The imaging device 2 captures an image of the culture vessel placed on the stage. The imaging device 2 irradiates light onto the cells cultured in the culture vessel, and captures the transmitted light that has passed through the cells as an image of the cells. As a result, images such as a transmission image, a phase contrast image, a dark field image, or a bright field image of the cells are acquired. Fluorescence emitted from the color-producing substance itself incorporated into the biological material, or fluorescence emitted by a substance having a chromophore binding to the biological material, may be captured as the image of the cells described above. As a result, the imaging device 2 acquires a fluorescent image such as a DAPI (4',6-diamidino-2-phenylindole) stained image or a Hoechst (registered trademark) stained image. The imaging device 2 or the display terminal 3 may have the function of the image processing device 1. The display terminal 3 is composed of a notebook computer, a panel computer, a tablet computer, or the like.

[0011] 2 is a block diagram showing an example of the hardware configuration of the image processing device 1. The image processing device 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 15, and a reading unit 16. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 15, and the reading unit 16 are connected by a bus B.

[0012] The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The control unit 11 reads out and executes a control program 1P (computer program) stored in the auxiliary storage unit 13, thereby realizing a functional unit that performs various information processing, control processing, etc. related to the image processing device 1.

[0013] The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data necessary for the control unit 11 to execute arithmetic processing.

[0014] The auxiliary storage unit 13 is a hard disk or a solid state drive (SSD) or the like, and stores a control program 1P and various databases (DBs) necessary for the control unit 11 to execute processing. The auxiliary storage unit 13 stores an input attribute DB 131, a result DB 132, a result detail DB 133, a first training data DB 134, and a second training data DB 135. The auxiliary storage unit 13 also stores a first learning model 141 and a second learning model 142. The auxiliary storage unit 13 may be an external storage device connected to the image processing device 1. The various DBs and the like stored in the auxiliary storage unit 13 may be stored in a database server or cloud storage different from the image processing device 1.

[0015] The communication unit 15 communicates with the imaging device 2 and the display terminal 3 via the network N. In addition, the control unit 11 may use the communication unit 15 to download the control program 1P from another computer via the network N or the like and store it in the auxiliary storage unit 13.

[0016] The reading unit 16 reads the portable storage medium 1a including a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the auxiliary storage unit 13. The control unit 11 may also download the control program 1P from another computer via a network N or the like and store it in the auxiliary storage unit 13. Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

[0017] 3 is a block diagram showing an example of the hardware configuration of the display terminal. The display terminal 3 includes a control unit 31, a main memory unit 32, an auxiliary memory unit 33, a communication unit 34, an input unit 35, a display unit 36, and an interface unit 37. Each component is connected by a bus B.

[0018] The control unit 31 has one or more arithmetic processing devices such as a CPU, an MPU, a GPU, etc. The control unit 31 reads out and executes a control program 3P stored in the auxiliary storage unit 33, thereby performing various information processing related to the display terminal 3.

[0019] The main memory unit 32 is an SRAM, a DRAM, a flash memory, etc. The main memory unit 32 mainly temporarily stores data necessary for the control unit 31 to execute arithmetic processing.

[0020] The auxiliary storage unit 33 is a hard disk or an SSD, etc., and stores the control program 3P and various data necessary for the control unit 31 to execute processing.

[0021] The communication unit 34 communicates with the image processing device 1 via the network N. In addition, the control unit 31 may use the communication unit 34 to download the control program 3P from another computer via the network N or the like, and store it in the auxiliary storage unit 33.

[0022] The input unit 35 is a keyboard and a mouse. The display unit 36 ​​includes a liquid crystal display panel or the like. The display unit 36 ​​displays the results of cell analysis, etc. The display unit 36 ​​and the input unit 35 may be integrated into one to form a touch panel display.

[0023] The interface unit 37 communicates with the imaging device 2 according to a serial communication standard such as USB (Universal Serial Bus). The display terminal 3 acquires cell images via the interface unit 37. Note that the imaging device 2 may have the functions of the display terminal 3.

[0024] Next, an outline of the analysis process performed by the cell analysis system 100 will be described. FIG. 4 is a flow chart showing an example of the procedure of the analysis process. The control unit 11 of the image processing device 1 acquires a plurality of images to be input (step S1). The plurality of images are images of the same sample captured under different conditions. The magnification, angle of view, etc., at the time of capturing each image are the same, and the cells are captured at the same position in each image. The plurality of images are, for example, a stained image A captured by staining the cells with stain A, a stained image B captured by staining the cells with stain B, and a bright field or phase contrast image. The plurality of images may be processed images in which these images are processed to emphasize specific tissues. These images are acquired directly from the imaging device 2, or the images acquired from the imaging device 2 are temporarily stored in the auxiliary storage unit 13, and then acquired from the auxiliary storage unit 13. The control unit 11 inputs the bright field or phase contrast image to the first learning model 141, and acquires a mask image output by the first learning model 141 (step S2). The mask image is a binarized image (binary image), in which the nuclear region of each cell is white and the rest is black. The control unit 11 uses the mask image to determine the category of each cell (step S3). The control unit 11 generates a partial image including one cell from each image, and inputs the generated partial images to the second learning model 142 to perform category determination (type determination). The control unit 11 ends the process. The category may indicate the state of the cell, such as whether it is alive or dead.

[0025] Next, the procedure of the analysis process will be described again. Fig. 5 to Fig. 7 are explanatory diagrams showing a part of the analysis process. Fig. 5 is an explanatory diagram of acquisition of a mask image. When a phase contrast image among a plurality of images is input to the first learning model 141, a mask image in which the nuclear region of each cell is white is obtained. The first learning model 141 is, for example, U-NET. The first learning model 141 may be a GAN (Generative Adversarial Network), SegNet, SSD (Single Shot Multibox Detector), etc.

[0026] FIG. 6 is an explanatory diagram of image cutting. The control unit 11 searches for an area including a nucleus area from the mask image, and generates a rectangular area 61 (in this embodiment, a rectangular area circumscribing the nucleus area) based on the search result. The control unit 11 searches for a white pixel from the upper left of the mask image (the origin of the image coordinates). If the control unit 11 determines that the search is successful, it determines a representative point (cx, cy) of the white area including the detected white pixel. cx is the x coordinate value of the representative point, and cy is the y coordinate value of the representative point. The representative point is the center of gravity of the white area or the center of the smallest rectangle that contains the white area. The control unit 11 determines a target area (x, y, w, h, cx, cy) that contains one cell (target cell) from the representative point. x and y are the x coordinate value and y coordinate value of the upper left corner of the target area. w and h are the width (length in the x-axis direction) and height (length in the y-axis direction) of the target area. Note that the target area to be generated is not limited to a rectangle, and may be a polygon, a circle, an ellipse, or the like. The target region to be generated may have any shape and size that includes the nuclear region and is assumed to include the target cell corresponding to the nucleus (the cytoplasm, in cases where the outer periphery of the cell is defined by the cytoplasm).

[0027] The control unit 11 cuts out an image 62 of each cell from the multiple input images based on the rectangular region 61. As described above, the multiple input images are stained image A (e.g., DAPI stained image), stained image B (Hoechst stained image), and a bright field or phase contrast image. The triplet of images labeled cell 1 and cell 2 shown at the bottom of FIG. 6 are examples of images cut out from stained image A, stained image B, and a bright field or phase contrast image. Here, the phase contrast image has areas other than the nuclear region masked. The image 62 on the right is the masked phase contrast image.

[0028] 7 is an explanatory diagram of category acquisition. Multiple images of each cell are input to the second learning model 142. The second learning model 142 outputs the probability of each category. The control unit 11 acquires the category with the highest probability as the cell category.

[0029] Next, the database used by the cell analysis system 100 will be described. FIG. 8 is an explanatory diagram showing an example of an input attribute DB. The input attribute DB 131 stores attributes of a plurality of captured images that are input. The input attribute DB 131 includes a sample ID column, an image type column, and an image file column. The sample ID column stores a sample ID that can uniquely identify a sample. The image type column stores the type of the acquired image. Phase difference indicates a phase contrast image. Staining A indicates stained image A, and staining B indicates stained image B. The image file column stores the file name of the image file that is the actual image. The image file is stored in the auxiliary storage unit 13 or the like. Although three images are input, two images or four or more images may be used.

[0030] FIG. 9 is an explanatory diagram showing an example of a result DB. The result DB 132 stores an overview of the results of the analysis process. The result DB 132 includes a sample ID column, a nucleus column, a category A column, a category B column, a category C column, and a category D column. The sample ID column stores the sample ID. The nucleus column stores the number of nuclei obtained by the search. The number of nuclei corresponds to the number of recognized cells. The category A column stores the number of cells determined to belong to category A. The category B column stores the number of cells determined to belong to category B. The category C column stores the number of cells determined to belong to category C. The category D column stores the number of cells determined to belong to category D. Here, there are four categories, but it may be a number other than four.

[0031] FIG. 10 is an explanatory diagram showing an example of the result details DB. The result details DB 133 stores details of the analysis processing results. The result details DB 133 includes a sample ID column, a cell ID column, a category column, a coordinate column, a basis image column, and a heat map column. The sample ID column stores a sample ID. The cell ID column stores a cell ID that uniquely identifies each cell. For example, the cell ID is a sequence number indicating the order in which the cell was found in the search. The category column stores the category of the cell obtained in the analysis processing. The coordinate column stores coordinate values ​​indicating the area (rectangular area) used when cutting out the image of each cell. For example, the coordinate values ​​(x, y) of the upper left corner, the width (w) of the rectangle, the height (h) of the rectangle, and the coordinate values ​​(cx, cy) of the representative point are stored. For example, the representative point is the center of gravity of the white area in the mask image, or the center of the smallest rectangle including the white area. The basis image column stores an image used to determine the category of the cell. The heat map column stores a heat map image. A heat map is an image that shows the degree of attention given to each part of an image when determining the category of a cell.

[0032] FIG. 11 is an explanatory diagram showing an example of the first training data DB. The first training data DB 134 stores training data used when generating or re-learning the first learning model 141. The first training data DB 134 includes an ID string, an input image string, and a mask image string. The ID string stores an ID that uniquely identifies the training data. The input image string stores images to be input to the first learning model 141. The mask image string stores mask images that are correct images.

[0033] FIG. 12 is an explanatory diagram showing an example of the second training data DB. The second training data DB 135 stores training data used when generating or relearning the second learning model 142. The second training data DB 135 stores an ID string, an input 1 string, an input 2 string, an input 3 string, and a label string. The ID string stores an ID that uniquely identifies the training data. The input 1 string, the input 2 string, and the input 3 string store images to be input to the second learning model 142, respectively. These images are images cut out for each target region, with input 1 storing a phase contrast image, input 2 storing a stained image A, and input 3 storing a stained image B. The label string stores the correct answer value of the category. The number of input strings may be increased or decreased according to the number of images to be input to the learning model.

[0034] Next, the first learning model 141 will be described. The first learning model 141 is, for example, U-NET, which is one of the models of semantic segmentation. U-NET is a type of FCN (Fully Convolutional Networks) and includes an encoder that performs downsampling and a decoder that performs upsampling. U-NET is a neural network that does not have a fully connected layer and is composed only of a convolution layer and a pooling layer.

[0035] FIG. 13 is an explanatory diagram of the generation of the first learning model 141. The control unit 11 of the image processing device 1 acquires an input image and a mask image which is a correct answer image. The control unit 11 randomly cuts out a region of a predetermined size from the input image. The cut-out image is called a cut-out input image. The input image is an image for extracting a nuclear region (in this embodiment, a bright field image or a phase contrast image is used, but a stained image may also be used). The predetermined size is, for example, a rectangular region of 256 pixels x 256 pixels. The control unit 11 cuts out a region of the predetermined size included in a region corresponding to the rectangular region from the mask image. The cut-out image is called a cut-out correct answer image. It is desirable that the predetermined size is a size slightly smaller than the cut-out input image. For example, the predetermined size is 192 pixels x 192 pixels, and the center position is aligned with the center position of the cut-out input image. The reason for making the size of the cut-out correct answer image smaller than that of the cut-out input image is that there is little data in the peripheral portion of the cut-out input image, and therefore the accuracy may decrease. To match the size of the ground-truth image to the input image, black pixels are added around the ground-truth image.

[0036] The cut-out input image is input to U-NET. U-NET is set to output an image with the nuclear area in white and the other areas in black, so it learns to make the output image closer to the cut-out correct image. The above process is repeated so that the entire input image is processed.

[0037] 14 is a flowchart showing an example of the procedure of the first learning model generation process. The control unit 11 of the image processing device 1 refers to the first training data DB 134 to acquire an input image and a correct answer image (step S11). The control unit 11 acquires setting data (step S12). The setting data is the size of the cut-out input image and the size of the cut-out correct answer image, and is stored in advance in the auxiliary storage unit 13. The setting data may be provided for each training data or for each cell contained in a sample.

[0038] The control unit 11 cuts out an image of a predetermined size (cut-out input image) from the input image (step S13). The control unit 11 cuts out an image of a predetermined size (cut-out correct image) from the correct image (step S14). The control unit 11 performs learning (step S15). The control unit 11 inputs the cut-out input image to U-NET, and adjusts the parameters of U-NET so that the output image approaches the cut-out correct image. The control unit 11 judges whether or not learning is to be terminated (step S16). For example, the control unit 11 stores the cut-out area and judges whether or not the entire input image has been processed. Alternatively, the control unit 11 determines the number of executions that is estimated to be sufficient for learning based on the relationship between the size of the input image and the size of the cut-out, and judges whether or not the number of executions has been reached. If the control unit 11 judges not to terminate learning (NO in step S16), the process returns to step S13 and continues the process. If the control unit 11 judges to terminate learning (YES in step S16), the control unit 11 judges whether or not there is unprocessed training data (step S17). When the control unit 11 determines that there is unprocessed training data (YES in step S17), the process returns to step S11 and continues. When the control unit 11 determines that there is no unprocessed training data (NO in step S17), the control unit 11 stores the parameters of U-NET that define the first learning model 141 (step S18) and ends the process.

[0039] FIG. 15 is an explanatory diagram of mask image generation. FIG. 15A is an explanatory diagram showing how an image is cut out from an input image. When an image of a predetermined region is cut out from an input image, the cut out regions located on the left and right are cut out so that the regions overlap. FIG. 15A shows how the first cut out region and the second cut out region overlap. FIG. 15B is an explanatory diagram showing output image processing. Since there is an overlap in the cut out regions, there is an overlap between the partial output image obtained from the first cut out input image and the partial output image obtained from the second cut out input image. For the overlapping portion, the region is divided in half and each half is used. In the first partial output image, the right half of the overlapping region is discarded. In the second partial output image, the left half of the overlapping region is discarded. FIG. 15C is an explanatory diagram showing how the entire output image is constructed. The remaining portions of the first partial output image and the second partial output image are combined to generate a part of the entire output image. By repeating the same process, the entire output image is generated. By performing this type of processing, even if the amount of data is insufficient, the peripheral parts of the output result obtained from the peripheral parts of the input image are discarded, making it possible to suppress a decrease in accuracy. Although only horizontal overlap is shown in Fig. 15, overlapping parts are also provided in the vertical direction.

[0040] FIG. 16 is a flowchart showing an example of a procedure of a mask image generation process. The control unit 11 of the image processing device 1 acquires settings (step S21). The settings include the size of the area to be cut out, the size of overlapping areas in the horizontal and vertical directions, etc. The control unit 11 cuts out the input image according to the settings (step S22). The control unit 11 inputs the cut-out image to the first learning model 141 (step S23). The control unit 11 acquires a partial output image output by the first learning model 141 (step S24). The control unit 11 stores the acquired partial output image in a temporary storage area (step S25). The temporary storage area is provided in the main storage unit 12 or the auxiliary storage unit 13. The partial output image is stored in association with information indicating a position in the entire image, coordinate values, sequence numbers, etc. so that the entire output image can be constructed. The control unit 11 determines whether or not to end the process (step S26). The control unit 11 can determine whether or not an area that has not been cut out remains in the input image. If the control unit 11 determines not to end the process (NO in step S26), it returns the process to step S22. If the control unit 11 determines to end the process (YES in step S26), it generates an entire output image from the multiple partial output images stored in the temporary storage area (step S27). The control unit 11 stores the generated entire output image (step S28) and ends the process.

[0041] In the mask image generation process, a plurality of regions of a predetermined size are set in an image of a plurality of cells, with adjacent regions overlapping each other by a predetermined area. A partial image is cut out from the captured image according to the set regions, and the cut out partial image is input to the first learning model 141 to obtain a partial output image. When synthesizing the partial output images according to the set regions, an overlapping region that overlaps with an adjacent region is divided into two, and partial images of the overlapping region close to the adjacent region are discarded, and the plurality of partial output images are synthesized to generate an entire output image.

[0042] Next, the second learning model 142 will be described. The second learning model 142 is, for example, a neural network such as a CNN (Convolution Neural Network). The second learning model 142 has an input layer that receives input of each of a plurality of target area images, an output layer that outputs a cell category, and an intermediate layer that extracts image features of the target area image.

[0043] The input layer has a plurality of neurons that accept input of pixel values ​​of each pixel included in the target region image, and passes the input pixel values ​​to the intermediate layer. The intermediate layer has a plurality of neurons, extracts features of the nuclear region in which the cell nucleus is shown, and passes them to the output layer. For example, the intermediate layer has a configuration in which a convolution layer that convolves the pixel values ​​of each pixel input from the input layer and a pooling layer that maps the pixel values ​​convolved in the convolution layer are alternately connected, and finally extracts image features while compressing the pixel information of the nuclear region. The output layer is a classifier related to SVM (Support Vector Machine), has a plurality of neurons, and outputs the probability that the cell belongs to each category based on the image features output from the intermediate layer.

[0044] In this embodiment, the second learning model 142 is described as being a CNN, but the second learning model 142 is not limited to a CNN and may be a learning model constructed with other learning algorithms, such as a neural network other than a CNN, a Bayesian network, or a decision tree. In addition, although three target area images are input to one CNN in FIG. 7, this is not limited to this. Three CNNs may be generated, each including a plurality of convolution layers and pooling layers, and each corresponding to one of the three target area images. In this case, each CNN outputs the feature amount of the image. The feature amount is passed to the output layer via a connection layer and a fully connected layer.

[0045] FIG. 17 is a flowchart showing an example of the procedure of the second learning model generation process. The control unit 11 of the image processing device 1 acquires training data stored in the second training data DB 135 (step S31). The control unit 11 extracts a plurality of input images and a label, which is a correct category, from the training data (step S32). The control unit 11 performs learning (step S33). The control unit 11 inputs a plurality of input images to the input layer of the CNN. The CNN outputs the probability of belonging to each category from the output layer after arithmetic processing in the intermediate layer. The probability output from the output layer is, for example, a probability value ranging from 0 to 1. The control unit 11 compares the output from the output layer with the label of the training data, i.e., the correct value, and optimizes the parameters used for the arithmetic processing in the intermediate layer so that the output value of the output node corresponding to the correct category approaches 1. The parameters are, for example, weights (coupling coefficients) between neurons, and the coefficients of the activation functions used in each neuron. Although the method of optimizing the parameters is not particularly limited, for example, the control unit 11 optimizes various parameters using an error backpropagation method. Note that a configuration may be adopted in which one value indicating a category is output from the output layer.

[0046] The control unit 11 determines whether or not there is unprocessed training data (step S34). If the control unit 11 determines that there is unprocessed training data (YES in step S34), the process returns to step S31, and learning is performed using the unprocessed training data. If the control unit 11 determines that there is no unprocessed training data (NO in step S34), the control unit 11 stores the CNN parameters that define the second learning model 142 in the auxiliary storage unit 13 (step S35), and ends the process.

[0047] FIG. 18 is a flowchart showing an example of the procedure of the judgment process. The judgment process corresponds to step S3 in FIG. 4. The control unit 11 of the image processing device 1 acquires settings related to the judgment process (step S41). The settings include multiple types of input images, the file name of the mask image, the definition of the representative point of the white area, the size of the image to be cut out, and the like. The multiple types of input images are images of the same sample taken under different conditions. The magnification, angle of view, and the like at the time of taking each image are the same, and the cells are taken at the same position in each image. The multiple types of input images are, for example, a stained image A (first stained image) taken after staining the cells with stain A (first stain), a stained image B (second stained image) taken after staining the cells with stain B (second stain), and a phase contrast image.

[0048] The control unit 11 searches for a white pixel from the top left of the mask image (the origin of the image coordinates) (step S42). The control unit 11 judges whether the search is successful or not (step S43). The search is successful when a white pixel is detected. The search is unsuccessful when the search of the entire image is completed without detecting any new white pixels other than the white pixels already detected. When the control unit 11 judges that the search is successful (YES in step S43), it sets a representative point (cx, cy) of the white area including the detected white pixel (step S44). cx is the x coordinate value of the representative point, and cy is the y coordinate value of the representative point. The representative point is the center of gravity of the white area or the center of the smallest rectangle that contains the white area, etc. The control unit 11 determines a target area (x, y, w, h, cx, cy) that contains one cell (target cell) from the representative point (step S45). x and y are the x coordinate value and y coordinate value of the upper left corner of the target area. w and h are the width (length in the x-axis direction) and height (length in the y-axis direction) of the target region. All areas other than the white area including the representative point are filled in black. Even if the mask image of the determined target region includes multiple white areas, all areas other than the white area including the representative point are filled in black. By masking non-target nuclei, only images of the cells to be determined are input to the second learning model 142.

[0049] The control unit 11 uses the determined region to cut out an image of the target cell from stained image A (step S46). The control unit 11 uses the determined region to cut out an image of the target cell from stained image B (step S47). The control unit 11 uses the determined region to cut out an image of the target cell from the phase contrast image (step S48). At this time, the image cut out from the phase contrast image may be masked (blackened) except for the nucleus region. By masking, the target nucleus becomes clear.

[0050] The control unit 11 inputs the cut-out images (images of the target area) to the second learning model 142 (step S49). The control unit 11 acquires the cell category based on the output of the second learning model 142 (step S50). The control unit 11 stores the result (step S51). The control unit 11 stores the information of the target area, the category, etc. in the result detail DB 133. The control unit 11 returns the process to step S42 and searches for the next white pixel. If the control unit 11 determines that the search has failed (NO in step S43), it tallies the number of processed target areas and the number of cells for each category, and stores them in the result DB 132 (step S52). The control unit 11 ends the process.

[0051] Two input images may be used for cell analysis. For example, two images, stained image A and stained image B, are used. In this case, a second learning model 142 that has been trained using stained image A and stained image B as input may be used. Since various combinations of input images are expected, a second learning model 142 is prepared according to the combination of input images. Then, a second learning model 142 according to the combination of input images is selected, and the input image is input to the selected second learning model 142. A stained image usually does not stain the cell itself, but rather stains a specific tissue contained in the cell. The cell nucleus, cytoplasm, or a specific protein in the cytoplasm is stained so as to be excited by a specific wavelength. Therefore, it is also possible to classify cells into multiple categories from a single stained image depending on the staining pattern. For example, it is possible to classify a specific protein as category A when it is unevenly distributed in the periphery, and as category B when it is not.

[0052] In addition, although deep learning is used to generate the mask image, this is not limited to the above. When generating a mask image from an image obtained by staining the nuclei with DAPI or Hoechst, a binarization process may be used. In addition, deep learning and binarization may be combined.

[0053] In this embodiment, it is possible to determine the category of a cell in an image obtained by photographing the cell, without detecting a cellular region.

[0054] (Modification) In order to show the basis of the judgment when displaying the judgment result, a heat map may be created and displayed together with the judgment result. The heat map is an image showing which part of the input image the second learning model 142 based its judgment on. The heat map is generated using known techniques such as Grad-CAM (Gradient weighted Class Activation Mapping), Guided Grad-CAM, CAM, Grad-CAM++, Guided Grad-CAM++, Saliency Map, Guided Backpropagation, Layer-wise Relevance Propagation, Integrated Gradients, Smooth Grad, DeepLIFT (Deep Learning Important FeaTures), etc.

[0055] Next, the display screen of the judgment result will be described. FIG. 19 is an explanatory diagram showing an example of the result display screen. The result display screen d01 includes an extracted image name d011, a stained A image name d012, a stained B image name d013, a result table d014, cell information d015, an input image area d016, a judgment basis d017, and an original image area d018. The extracted image name d011 displays the name of the image that is the source of the mask image created to extract the nuclear area, for example, the image file name. The stained A image name d012 displays the name of the stained image A. The stained B image name d013 displays the name of the stained image B. The result table d014 displays an overview of the judgment result. The result table d014 is based on the contents stored in the result DB 132. The cell information d015 displays the judgment result for each cell. The input image area d016 displays one of the input images. In the input image area, when the mouse pointer is brought close to one cell nucleus, detailed information about the cell including the cell nucleus is displayed. A rectangle indicating the target region is displayed superimposed on the input image region d016. The category, cell ID, coordinate values ​​indicating the rectangular region, and coordinate values ​​of the representative point are displayed in cell information d015. An image indicating the basis is displayed in judgment basis d017. In FIG. 19, the upper image of judgment basis d017 is a phase contrast image in which areas other than the nuclear region are masked. The lower image of judgment basis d017 is a heat map. An image cut out from the input image is displayed in the original image region d018. The upper part of the original image region d018 displays a phase contrast image, the middle part displays a stained A image, and the lower part displays a stained B image.

[0056] FIG. 20 is a flow chart showing an example of the procedure of the result display process. The result display process is a process for generating the result display screen shown in FIG. 19. The control unit 31 of the display terminal 3 acquires a sample ID for which the result is to be displayed based on a user's instruction (step S71). The control unit 31 transmits the sample ID to the image processing device 1 (step S72). The control unit 11 of the image processing device 1 receives the sample ID from the display terminal 3 (step S73). The control unit 11 acquires the input attribute and the analysis result corresponding to the acquired sample ID (step S74). The control unit 11 acquires the input attribute from the input attribute DB 131 and the analysis result from the result DB 132. The control unit 11 generates a result display screen (step S75). The control unit 11 transmits the generated display screen to the display terminal 3 (step S76). The control unit 31 of the display terminal 3 receives the display screen and displays it on the display unit 36 ​​(step S77). The control unit 31 determines whether the input image area is pointed to by the mouse pointer or the like (step S78). The pointing includes a mouse over, a mouse click, etc. When the control unit 31 determines that the input image area is pointed (YES in step S78), it transmits the pointed coordinates to the image processing device 1 (step S79). The control unit 11 of the image processing device 1 receives the coordinate values ​​(step S80). The control unit 11 determines whether or not there is a rectangular area including the point indicated by the coordinate values ​​(step S81). The control unit 11 compares the received coordinate values ​​with the coordinate column values ​​in the result details DB 133 to make this determination. When the control unit 11 determines that there is a rectangular area including the point indicated by the coordinate values ​​(YES in step S81), it acquires the result details from the result details DB 133 (step S82). The result details include a determination basis image and a heat map. The control unit 11 acquires multiple original images used in the category determination based on the rectangular area (step S83). The control unit 11 generates a result display screen including the result details and the original images (step S84). The control unit 11 moves the process to step S76. If the control unit 11 determines that there is no rectangular area including the point indicated by the coordinate values ​​(NO in step S81), the control unit 11 moves the process to step S84.

[0057] If the control unit 31 determines that the input image area is not pointed to by the mouse pointer or the like (NO in step S78), it determines whether or not to end (step S85). If the control unit 31 determines that the processing should not end (NO in step S85), it repeats step S78. There are operations that can be performed other than area pointing and ending (e.g., changing the sample to be displayed), and when such an operation is instructed, the control unit 31 may perform the instructed processing. If the control unit 31 determines that the processing should end (YES in step S85), it ends the processing.

[0058] In the above description, the display terminal 3 transmits the coordinate values ​​of the point in the input image area to the image processing device 1, and the control unit 11 of the image processing device 1 determines whether or not there is a rectangular area that includes the point indicated by the coordinate values, but this is not limited to the above. The image processing device 1 may generate a critical map indicating the rectangular area, and set the generated critical map in the input image area. In this case, the control unit 31 of the display terminal 3 determines whether or not there is a rectangular area that includes the pointed point, based on the critical map. If the control unit 31 determines that there is a rectangular area, it transmits information identifying the rectangular area (e.g., a cell ID) to the image processing device 1.

[0059] In this modified example, it is possible to refer to the basis for the judgment made by the second learning model 142.

[0060] 21 is a block diagram showing an example of functional units included in an image processing device. The image processing device 1 includes, as functional units, an identifying unit 11a, an extracting unit 11b, an output unit 11c, a generating unit 11d, and an acquiring unit 11e. Each of these functional units is realized by the control unit 11 operating based on a control program 1P.

[0061] The identification unit 11a identifies a target region including the nucleus of a target cell based on an image of multiple cells. The extraction unit 11b extracts an image of the target region from each of multiple types of images including an image and an image of a different type from the image. The output unit 11c identifies and outputs the type of the target cell based on the extracted target region image. The generation unit 11d generates a mask image that masks the region of the target cell other than the nucleus of the target cell in the target region. The acquisition unit 11e inputs the target region image extracted from the multiple types of images to a learning model that has been trained to output the type of the target cell when the target region image is input, and acquires the type of the target cell from the learning model.

[0062] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0063] 100 Cell Analysis System 1 Image processing device 11 Control section 11a Specific part 11b Extraction part 11c Output section 11d Generator 11e Acquisition Department 12 Main memory 13 Auxiliary storage 131 Input Attribute DB 132 Results DB 133 Result details DB 134 1st training data DB 135 Second training data DB 141 First Learning Model 142 Second Learning Model 15 Communications Department 16 Reading unit 1P control program 1a Removable storage media 1b Semiconductor memory 2. Imaging device 3. Display terminal N Network

Claims

1. Extracting a plurality of partial images of a predetermined size from a first image of a plurality of cells by overlapping adjacent partial images; A first learning model is trained to generate a mask image in which an area other than the nucleus of a cell in an image containing a cell is masked when the image is input, and the extracted partial images are input to generate partial mask images in which an area other than the nucleus of a cell in the partial images is masked, discarding half of the area of ​​the overlapping portions of the generated partial mask images on the side of adjacent partial mask images, and then synthesizing the multiple partial mask images to generate a mask image; Identifying a target region including a nucleus of a target cell in the first image based on the generated mask image; The identified target region is extracted from each of a plurality of types of images including the first image and a second image corresponding to the first image and obtained by photographing the plurality of cells under conditions different from those of the first image; A second learning model is trained to output the type of target cell in a target region when multiple images of the target region are input, and multiple images of the target region extracted from each of the multiple types of images are input to identify and output the type of the target cell in the target region. A computer program that causes a computer to carry out processing.

2. The plurality of types of images include at least two of a first stained image stained with a first staining agent, a second stained image stained with a second staining agent, a bright field image, or a phase contrast image.

2. The computer program product of claim 1.

3. The target region is specified by the coordinate values ​​of a rectangular region including the nucleus of the target cell and the coordinate values ​​of a representative point representing the position of the target cell.

3. A computer program according to claim 1 or 2.

4. Outputting the type of target cell and an image of the target region A computer program product according to any one of claims 1 to 3, which causes a computer to execute a process.

5. Outputting an image showing the type of the target cell and the basis for determining the type. A computer program product according to any one of claims 1 to 4, which causes a computer to execute a process.

6. In an output device having a control unit, The control unit: Extracting a plurality of partial images of a predetermined size from a first image of a plurality of cells by overlapping adjacent partial images; A first learning model is trained to generate a mask image in which an area other than the nucleus of a cell in an image containing a cell is masked when the image is input, and the extracted partial images are input to generate partial mask images in which an area other than the nucleus of a cell in the partial images is masked, discarding half of the area of ​​the overlapping portions of the generated partial mask images on the side of adjacent partial mask images, and then synthesizing the multiple partial mask images to generate a mask image; Identifying a target region including a nucleus of a target cell in the first image based on the generated mask image; The identified target region is extracted from each of a plurality of types of images including the first image and a second image corresponding to the first image and obtained by photographing the plurality of cells under conditions different from those of the first image; A second learning model is trained to output the type of target cell in a target region when multiple images of the target region are input, and multiple images of the target region extracted from each of the multiple types of images are input to identify and output the type of the target cell in the target region. Output device.

Images