Image processing method, drug sensitivity test method, and image processing device
通过深度学习技术生成带虚拟标记的图像并检测细胞代表点位置,解决了未标记细胞图像中自动计测问题,降低了实验成本并适用于观察细胞变化。
Patent Information
- Application Number
- CN201980031660.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-10
- Filing Date
- 2019-05-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-05-17
AI Technical Summary
It is difficult for the prior art to automatically measure the location and number of cells in images of unlabeled cells, especially in innovative drug screening experiments, labeling introduction increases the cost of experiments and is not suitable for continuous observation.
Two learning models are established through deep learning: the first learning model generates images with virtual marks, and the second learning model detects the representative point positions of specific parts to achieve high-precision processing of unlabeled images.
High-precision detection of unlabeled cell images is achieved, and the cell position and number are automatically measured, which reduces the experimental cost and is suitable for experiments that observe the changes in cells over time.
Smart Images

Figure CN112135912B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing method for determining the position of a specific site in a cell included in an image obtained by photographing the cell, a drug sensitivity test method using the image processing method, and an image processing device. Background Art
[0002] For example, in experiments aimed at screening innovative drugs, observing the changes in cells in a sample into which a drug has been put is widely performed. As previous test methods, the ATP method, the MTT method, etc. are known. In addition, in recent years, the number, size, shape, etc. of cells are automatically measured based on the image obtained by shooting the sample. In such experiments, a marker selectively manifested in a part (e.g., a cell nucleus) with a specific property is introduced into the cell (this operation is also referred to as labeling). Then, the manifested part of the marker in the image is detected by image processing (e.g., refer to patent document 1).
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-521037. Summary of the invention
[0006] Problem that the invention aims to solve
[0007] Such labeling introduction will change the cells to be observed. Therefore, it may not be suitable for continuous cell observation. In addition, the reagent used to give the label and its introduction operation become the reason for the increase in experimental costs. Therefore, it is expected to identify cells by images without using labels.
[0008] Especially in experiments for innovative drug screening, it is expected to easily understand how many living cells there are among various objects contained in an image. However, images taken without using markers are suitable for observing the shape and texture of cells, but may not be suitable for automatically counting the position and number of cells. Therefore, it is expected to establish a technology for automatically counting the position and number of cells based on images without using markers.
[0009] Means used to solve problems
[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing technology capable of automatically measuring the positions and the number of cells included in an image even from an image without using a marker.
[0011] One embodiment of the present invention is an image processing method for detecting the position of a specific detected part based on a test image obtained by photographing cells, wherein, in order to achieve the above-mentioned purpose, the test image is input into a first learning model, the output image of the first learning model is input into a second learning model, and the output image of the second learning model is output as a result image in which the position of the detected part is indicated by its representative point.
[0012] In addition, another embodiment of the present invention is an image processing device that detects the position of a specific detected part based on a test image obtained by photographing cells, wherein, in order to achieve the above-mentioned purpose, the image processing device has: an image acquisition unit that acquires a bright field image or a phase difference image as the test image; and an image processing unit that inputs the test image into a first learning model, inputs the output image of the first learning model into a second learning model, and outputs the output image of the second learning model as a result image in which the position of the detected part is indicated by its representative point.
[0013] In these inventions, the first learning model is established by performing deep learning (deep learning) by using training data that associates images of cells that are identical to each other, that is, a first image and a second image, so that the second image corresponds to an input and the first image corresponds to an output. The first image is an image in which a mark corresponding to the detected part appears, and the second image is an image in which the mark does not appear. In addition, the second learning model is established by using a third image in which an image of a cell, that is, an image in which the mark appears, and information indicating the position of the representative point contained in the third image, as training data, so that the third image corresponds to an input and the position of the representative point corresponds to an output.
[0014] In the present invention thus constituted, the position of the detection site having specific characteristics can be detected with high accuracy even from a test image without an introduced marker. The reason for this is as follows.
[0015] In the present invention, the first learning model is a learning model established by performing deep learning using training data that associates a first image in which a mark corresponding to the detected part is displayed and a second image in which the mark is not displayed. Therefore, in the first learning model after learning, the marked image and the unmarked image are associated with each other.
[0016] Therefore, if a test image in which the mark is not visible is input to the first learning model, an image with a mark corresponding to the test image is obtained as an output. This image is equivalent to an image obtained when a mark is introduced into a sample used for shooting the test image and the sample is photographed under the condition that the mark is visible. That is, the first learning model has a function of generating an image with a virtual mark, such as an image obtained by photographing the same sample with the mark visible, from a test image without a mark. Thus, even if the mark is not actually introduced into the sample, an image as if the mark is visible can be obtained.
[0017] On the other hand, the second learning model is a learning model established by performing deep learning using training data that associates the image where the marker appears with information indicating the position of the representative point of the detected part. Therefore, in the second learning model after learning, the image where the marker appears and the position of the representative point of the detected part contained therein are associated with each other.
[0018] Therefore, when the image in which the marker is displayed is input to the second learning model, an image indicating the position of the representative point of the detected part existing in the input image is obtained as its output. Here, if the output image of the first learning model is used as the input image, the final output image in which the position of the representative point of the detected part in the image is determined is obtained. The output image of the first learning model is an image with virtual markers generated from a test image without markers. Therefore, through the above series of processing, the position of the representative point of the detected part in the image can be determined from the test image without markers.
[0019] In the past, images of cells were classified by machine learning with training. However, most of these methods are based on clustering in the feature space based on artificially selected feature quantities. In this case, which feature quantity to use will affect the classification accuracy, so it is necessary to determine the feature quantity based on the characteristics shown in the detected part or other parts. Therefore, specialized knowledge for selecting feature quantities is required in operation. In addition, the results of learning are not necessarily universal for various cells.
[0020] In contrast, in the present invention based on deep learning, there is no need to select feature quantities in advance. As long as sufficient image samples can be prepared as training data, a high-precision learning model can be established. Therefore, the same learning algorithm can be used to deal with various cells with different features shown in the image.
[0021] Thus, in the present invention, the position of the representative point representing the detected part can be determined with high accuracy. Therefore, even in a test image without a mark, the position and number of the detected parts in the image can be automatically measured.
[0022] In addition, another embodiment of the present invention is a drug sensitivity test method, comprising: a process of obtaining an image obtained by photographing a sample in which the drug serving as an evaluation object is administered to cultured cells; a process of using the acquired image as the test image and counting the number of living cells serving as the detection site using the above-mentioned image processing method; and a process of determining the sensitivity of the cells to the drug based on the counting result.
[0023] In the invention thus constituted, the above-mentioned image processing method is performed using the living cells in the test image as the detected parts. Thus, the positions where the living cells exist in the test image can be determined with high accuracy. Using the results, the living cells in the test image can be selectively detected, and their number can be calculated with high accuracy. Therefore, based on the number of living cells, the sensitivity of the cells to the drug can be effectively determined.
[0024] Effects of the Invention
[0025] According to the present invention, by using a learning model established by deep learning in advance, the positions of representative points of the detected parts in the image can be detected with high accuracy from an unlabeled test image, thereby outputting a result image suitable for automatically measuring the positions and numbers of the detected parts.
[0026] The above and other objects and novel features of the present invention will become more apparent by reading the following detailed description with reference to the accompanying drawings, which are, however, for illustration only and do not limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a diagram showing the concept of an image processing process according to one embodiment of the present invention.
[0028] Figure 2 is a flowchart showing the image processing process.
[0029] Figure 3 is a flowchart showing a learning process for establishing a first learning model.
[0030] Figure 4 It is a schematic diagram showing the operation of the learning model adopted in this embodiment.
[0031] Figure 5 is a flowchart showing the learning process of establishing the second learning model.
[0032] Fig. 6A This is a diagram illustrating a method of assigning representative points.
[0033] Figure 6B This is a diagram illustrating a method of assigning representative points.
[0034] Figure 7 This is a block diagram showing an example of a configuration of a device capable of executing the image processing according to the present embodiment.
[0035] Figure 8 This is a diagram showing an example of the measurement result of the number of cells according to the present embodiment.
[0036] Fig. 9A This is a diagram comparing the cell number measurement results of this embodiment with the ATP analysis method.
[0037] Fig. 9B This is a diagram comparing the cell number measurement results of this embodiment with the ATP analysis method.
[0038] Fig.10 The diagram shows an example of an image of a triple-stained sample.
[0039] Fig.11 The diagram shows an example of an image of a triple-stained sample.
[0040] Fig.12 This is a diagram showing an example of experimental results of comparing counting results.
[0041] Fig.13 This is a flowchart showing a drug sensitivity test method to which the image processing method according to the present embodiment is applied.
[0042] Fig.14 It is a figure which shows another embodiment of the present invention. DETAILED DESCRIPTION
[0043] Figure 1 FIG. 1 is a diagram showing the concept of an image processing process according to an embodiment of the present invention. Figure 2 : is a flowchart showing the image processing process. The purpose of the image processing process is to detect the position of a specific detection part (for example, a cell nucleus) based on a test image It captured containing cells, and finally to count the number of cells that meet the specified conditions. More specifically, the test image It used as the object is, for example, a bright field image or a phase difference image captured by an optical microscope. In addition, the detection result of the detection part is output as a result image Io indicating the position of a representative point of the detection part.
[0044] The specific processing process is summarized as follows. First, a test image It is acquired as a target (step S101). The test image It can be acquired by performing a new shooting. Alternatively, the test image It can be acquired by reading image data that has been previously shot and stored in a suitable storage device.
[0045] The test image It is input to the first learning model 10 (step S102). The first learning model 10 outputs an intermediate image Im created based on the test image It (step S103). The intermediate image Im is an image obtained by photographing the same sample as the test image It under the condition that the marker is visible.
[0046] That is, the first learning model 10 has a function of creating an intermediate image Im with virtual labels from an unlabeled test image It. In this sense, the first learning model 10 is also referred to as a "label generation model", for example. If an image with virtual labels can be created with high accuracy (i.e., in a manner that has a high degree of similarity to an image captured by actually introducing labels) from an unlabeled image, it is not necessary to introduce labels into the sample. This eliminates the need for invasive imaging of the cells to be observed and also helps reduce the cost associated with the introduction of labels.
[0047] In the shooting without introducing the marker, there is no need to perform the processing for introducing the marker on the sample. Therefore, the cells can be photographed in a non-invasive manner, and it can also be used for the purpose of observing the changes of cells over time. In addition, since there is no need for a reagent as a marker and the processing for introducing the marker, the cost of the experiment including the shooting can also be suppressed. In addition, depending on the type of marker, it takes a long time, such as several days, until the shooting can be done, but such time can also be omitted. Therefore, it can also be applied to the observation of samples whose cell states change significantly in a short period of time, for example, it can also be applied to the field of regenerative medicine.
[0048] The unlabeled images captured in this way, such as bright field images of cells, clearly show the shape of the cells, the shades of the internal structure, etc., and are therefore suitable for visual observation. On the other hand, for the purpose of quantitatively and automatically detecting specific parts that require attention, such as the position and number of cell nuclei in the image, labeled images are still advantageous in terms of measurement accuracy. Therefore, if it is possible to generate labeled images from unlabeled images, high-precision quantitative measurements can also be performed based on unlabeled images.
[0049] For this purpose, the first learning model 10 is established by deep learning using pre-collected training data. The training data is a combination of a plurality of first training images I1 and a second training image I2 obtained by respectively capturing images of the same position of a sample prepared in advance including cells into which a marker has been introduced. Here, the first training image I1 is obtained by capturing the sample with the marker visible, and the second training image I2 is obtained by capturing the sample without the marker visible.
[0050] As a marker, for example, a marker that selectively emits fluorescence at a specific part of a cell can be used. In this case, a fluorescent image of a sample photographed under excitation light illumination can be used as a first training image I1, and a bright field image of the same sample photographed under visible light illumination can be used as a second training image I2. By comparing corresponding positions in the images between the first training image I1 and the second training image I2, the representation of the same object in the sample when photographed with a marker and when photographed without a marker can be correlated with each other.
[0051] A plurality of such examples are collected and used as training data to perform machine learning. In this way, for example, it is possible to infer how an object appearing in an unmarked image would appear if the object was photographed with the markers showing. By utilizing this, it is possible to create an image with virtual markers from an unmarked image. In particular, when deep learning is used as a learning algorithm, it is not necessary to manually provide feature quantities for analyzing images. Therefore, there is no need for specialized knowledge for appropriately selecting feature quantities according to the intended use. Furthermore, it is possible to establish an optimal learning model that eliminates the possibility of misjudgment due to inappropriate selection of feature quantities.
[0052] Various well-known materials already exist about the principles and learning algorithms of deep learning, so detailed description is omitted here. The deep learning method available in this embodiment is not limited to a specific algorithm. In addition, as a deep learning method that can be particularly preferably used in this embodiment to use paired data such as images for input and output to learn the correlation between paired data, for example, there is a method called "pix2pix" based on Conditional GAN (reference: Phillip Isola et al., Image-to-Image Translation with Conditional AdversarialNetworks.CVPR, 21-Nov.2016, URL: https: / / arxiv.org / pdf / 1611.07004v1.pdf).
[0053] Figure 3 2 is a flowchart showing a learning process for establishing a first learning model. As described above, by collecting a plurality of combinations of bright field images and fluorescent images obtained by photographing the same position of the same sample (step S201), and performing deep learning using them as training data, the first learning model 10 can be established (step S202). Preferably, the training images (the first training image I1 and the second training image I2) are obtained by photographing cells of the same type as the cells of the detection site noted in the test image It.
[0054] Figure 4 Schematic diagram showing the operation of the pix2pix algorithm of the learning model adopted in this embodiment. Figure 4 As shown, the first learning model 10 includes an image generation unit 11, a recognition unit 12, and an optimizer 13. The image generation unit 11 includes an encoder 11a that captures image features using a plurality of convolutional layers; and a decoder 11b that generates images by performing inverse operations based on the features using a deconvolutional layer having the same number of layers. A second training image I2, which is a bright field image, is provided to the image generation unit 11 as an input image. The recognition unit 12 recognizes whether it is an input image or a generated image based on the image features. The first training image I1, which is a fluorescent image, is input to the recognition unit 12 to perform real object learning. On the other hand, the generated image Ig output from the image generation unit 11 is input to the recognition unit 12 to perform pseudo object learning. The optimizer 13 adjusts the internal parameters of the learning model so that the generated image Ig output from the image generation unit 11 is close to the real object (first training image I1). Learning is performed by repeating the above-mentioned actions. If learning is sufficient, it has the function of generating a virtual fluorescent image (intermediate image Im) corresponding to the unknown bright field image (test image It).
[0055] In addition, here, fluorescent labeling is used as a method of labeling cells, but the labeling method is not limited to this. For example, it is also possible to selectively dye a specific part with a suitable dye. In this case, a combination of an image of an unstained sample and an image of a stained sample is used as training data for deep learning. In this way, it is possible to infer and create an image obtained when the sample is stained based on the newly provided unstained image.
[0056] In addition, multiple markers can be introduced into one sample. For example, Calcein is known as a marker that appears green in the cytoplasm of living cells. In addition, Propidium Iodide (PI) is known as a marker that appears red in the nucleus of dead cells. Introducing them into the same sample is also widely done. In such an example, various markers appear in different colors in the fluorescent image. Therefore, for example, if it is a color image, living cells and dead cells can be distinguished by color decomposing the image.
[0057] This method can also be introduced in the image processing of the present embodiment. That is, deep learning is performed using a combination of a fluorescent image and a bright field image of a sample into which these two markers are introduced as training data. In this way, an intermediate image is obtained in which objects corresponding to living cells among objects in the bright field image as a test image are displayed in green corresponding to calcein, and objects corresponding to the nuclei of dead cells are displayed in red corresponding to PI.
[0058] In addition, when a plurality of marks with different luminous colors are introduced in this way, at least the marked image needs to be an image that can be processed by distinguishing colors. For example, data obtained by shooting a full-color image and decomposing the images into RGB colors can be used. In addition, image data shot in monochrome through a bandpass filter corresponding to the luminous color can also be processed as virtual version image data corresponding to the luminous color. For example, if a high-sensitivity cooled CCD camera is used for monochrome shooting, image data with high resolution and capable of color reproduction can be obtained.
[0059] In this way, deep learning is performed by combining the labeled image and the unlabeled image as training data to establish the first learning model 10. The first learning model 10 established in this way generates an image with virtual labels by analogy with the fluorescent image obtained when the label is introduced into the same sample based on the new bright field image without label, that is, the test image It. This image is the intermediate image Im.
[0060] Figure 1 The intermediate image Im shown is an image with virtual markers corresponding to the test image It. Here, among the objects included in the test image It, the objects in which two types of markers appear respectively are represented with two densities.
[0061] return Figure 1 as well as Figure 2 Let's continue the description of the image processing process. Through the processing so far, it is possible to create an image with a virtual mark based on the image obtained by shooting a sample without introducing a mark. By selecting a suitable mark corresponding to the target detected part, the presence and position of the detected part in the image can be displayed on the image. For example, if the test image It and the intermediate image Im are displayed overlapping, it is possible to simply visually confirm which of the objects displayed in the test image It corresponds to the detected part. For example, when visually observing the cells in the image, the life and death of each cell can be clearly displayed. Objects that do not correspond to the mark, that is, objects that are not the objects of observation, are removed from the image at this moment and will not affect subsequent processing.
[0062] In a labeled image, especially a fluorescent image in which the label emits light in response to the excitation light, in principle, the information of the shape of the original object is often lost. Therefore, in the image with virtual labels output by the first learning model 10, sometimes information related to the correct shape of cells, etc., width in the image, etc. is not obtained. For example, in the case where multiple cells are in contact or close to each other in the sample, they may appear as one in the image. This may be an error factor in quantitatively detecting the position and number of cells.
[0063] Therefore, in this embodiment, in order to determine the position of the detection part based on the intermediate image Im produced by the first learning model 10, the second learning model 20 is used. Figure 1 As shown, the second learning model 20 is established by performing deep learning by combining the third training image I3 and the position information image Ip prepared in advance as training data. The position information image Ip is an image showing the position of a representative point representing the detected part in the corresponding third training image I3.
[0064] As described later, the second learning model 20 thus established has a function of detecting the position of a representative point representing a detected part in an image based on a labeled image. In this sense, the second learning model 20 is also referred to as a "positioning model". In the specific processing process, as described above, an intermediate image Im, an image with virtual labels corresponding to the test image It, is generated by the first learning model 10, and the intermediate image Im is input to the second learning model 20 (step S104).
[0065] The second learning model 20 creates an image representing the position of the representative point of the detected part based on the input intermediate image Im, and outputs it as a result image Io (step S105). For example, if the detected part is regarded as a "living cell" and the center of gravity position of the nucleus of the living cell is regarded as a representative point, the result image Io becomes an image representing the "center of gravity position of the nucleus of the living cell" in the object contained in the test image It. The result does not necessarily have to be output as an image like this, for example, it can also be output as data representing the coordinates of the detected representative point. The definition of the detected part and the representative point is not limited to the above content, and is arbitrary.
[0066] If data for determining the positions of representative points of the detected parts in the image can be obtained in this way, it is easy to automatically count the number of detected parts based on the data (step S106). If the counting result is output and presented to the user (step S107), the user can obtain the counting result of the detected parts contained in the test image It by simply preparing the test image It.
[0067] Figure 5 is a flowchart showing the learning process for establishing the second learning model. Figure 1 The third training image I3 shown in FIG. 1 is a plurality of fluorescence images (step S301). The fluorescence image in this case may be any one of the following (1) and (2):
[0068] (1) the same fluorescence image as the image collected as the first training image I1,
[0069] (2) A fluorescence image captured separately from the first training image I1.
[0070] Furthermore, when the virtual labeled image created by the first learning model 10 shows sufficiently high similarity with the actual fluorescence image, the following (3) can also be used:
[0071] (3) A virtual fluorescence image output by the first learning model 10 .
[0072] Alternatively, they may be combined to serve as training images.
[0073] In this case, the training image is a fluorescence image. However, in a broader sense, the training image may be an image in which the same detection site as the image with virtual markers output by the first learning model 10 is represented by the same marker.
[0074] With respect to the detected parts contained in the collected fluorescent image, information (position information) for determining the position of the representative point is collected (step S302). For example, the detected parts of a specific type of cell, a structure in a cell (e.g., a cell nucleus), etc. have a certain width in the image, and their shapes vary greatly from one individual to another. Therefore, it is difficult to uniquely represent the position of the detected part with simple data. Therefore, its representative point is appropriately determined, and the position of the detected part is represented by the position of the representative point.
[0075] For example, when the detected part is a relatively small structure such as a cell nucleus or mitochondria in a cell, a suitable position (for example, the center of gravity position) of the structure in the image can be used as a representative point. In addition, when individual cells are used as the detected part, the center of gravity position of their nuclei can also be used as a representative point. In this way, for example, even if multiple cells that are close to each other become one in the fluorescent image, if the position of the nucleus is determined, it includes multiple nuclei, and therefore, they are represented as separate cells. That is, multiple cells that appear to be one in the fluorescent image can be distinguished and can be counted separately.
[0076] In order to be able to perform the above-mentioned distinction and counting, the position of the representative point is set for the detected part contained in each collected fluorescent image. For example, the representative point can be set by displaying a fluorescent image as a typical example on the screen and having a skilled person specify the position of the input representative point. In addition, for example, the center or center of gravity of a relatively small structure indicated by a mark, such as a cell nucleus, can be obtained through image processing, and its position can be used as the position of the representative point. In addition, it is also possible to perform a known image processing (corrosion processing) that causes the object contained in the fluorescent image to shrink in stages, and use the point that remains at the end as the representative point. Such image processing that selects a representative point from an area with a width can appropriately apply the known MaxPooling processing, for example.
[0077] The combination of the fluorescence image collected in this way and the position information of the representative points of the detected parts contained therein is used as training data (step S303). Figure 1 In FIG. 1 , the position information is represented as an image Ip on which the position of the representative point is mapped. However, this is not limited to this, and for example, the position information may be represented as a table describing the position coordinates of the representative point.
[0078] The second learning model 20 is established by performing deep learning based on the prepared training data (step S304). For example, the third training image I3 as a fluorescent image is made to correspond to the input, and the position information image Ip indicating the position of the representative point is made to correspond to the output for learning. In this way, the position of the detected part can be detected based on the fluorescent image in which the mark corresponding to the detected part appears.
[0079] That is, the second learning model 20 established as described above has the following function: when the intermediate image Im output by the first learning model 10 or the newly acquired unknown fluorescence image is input, an image representing the detected part in the image by the position of the representative point is output. This output image becomes the result image Io. In addition, as the purpose of processing, it is sufficient to determine the position of the detected part. In this sense, instead of or in addition to making the result image Io representing the position of the representative point, for example, a table or list representing the position coordinates of the detected representative point may be made.
[0080] Fig. 6A as well as Figure 6B is a diagram illustrating a method of assigning representative points. Fig. 6A As shown in FIG. 1 , an object Q1 having a relatively simple shape can be considered to correspond to an independent cell (or cell mass). Therefore, a representative point P1 can be set inside the object. Figure 6B As shown, when the outer periphery of the object Q2 has a complex shape, it can be considered that a plurality of cells are partially in contact with each other. Therefore, a plurality of representative points P2 and P3 can be set according to the shape.
[0081] Such cell connections can be addressed by preparing training data that collects various examples and establishing the second learning model 20 through deep learning. That is, when objects of the same shape appear in an unknown fluorescent image or an intermediate image Im output by the first learning model 10, they are not mistaken for a single cell, but their respective representative points are detected corresponding to each cell.
[0082] Thus, the second learning model 20 is used for the purpose of determining the position of the representative point of the detected part based on the intermediate image Im output from the first learning model 10 in this embodiment. In addition, the second learning model 20 can also be used for the purpose of detecting the position of a specific part based on a fluorescent image in general. For example, the second learning model 20 can be used for automatically measuring the position and number of specific parts such as cells based on the fluorescent image of a sample into which a fluorescent marker is introduced. The learning algorithm is the same as that of the first learning model 10 described above.
[0083] As described above, the shape of cells may not be clearly visible in the fluorescence image. This may cause the following problems: it is difficult to automatically measure the position and number of cells based on the fluorescence image, and it is difficult to improve the accuracy of the measurement results. The second learning model 20 can solve this problem.
[0084] In addition, as described later, with respect to the learning model for evaluating samples that act on drugs that affect cells, it is desirable to establish multiple images obtained by photographing samples prepared at various drug concentrations as training images. The morphological characteristics of cells change due to the action of the drug, and the magnitude of the change depends on the drug concentration. Therefore, for example, a learning model established using only images of samples with low drug concentrations as training images is not necessarily effective for images of samples with high drug concentrations. The opposite is also true. When a group of images in which images of samples with various drug concentrations are mixed is used as a training image, the established learning model can also cope with samples with various drug concentrations.
[0085] Figure 7 1 is a block diagram showing an example of a device structure capable of performing image processing of the present embodiment. The device is an example of an image processing system capable of performing the image processing method of the present invention. The image processing system 100 includes an image processing device 110 and a photographing device 120. The image processing device 110 is an embodiment of the image processing device of the present invention, and includes a CPU (Central Processing Unit) 111 having an image processing unit 112 built therein, a memory 113, an interface 114, and a display unit 115.
[0086] The CPU 111 executes the above-mentioned image processing process and deep learning by executing a predetermined control program pre-stored in the memory 113. The processing of image data in this process is performed by the image processing unit 112. The memory 113 stores the control program executed by the CPU 111 and the data generated by the processing. The interface 114 is responsible for data communication with external devices. In addition, the interface 114 receives operation input from the user via an input device (for example, a keyboard, a mouse, etc.) not shown in the figure. The display unit 115 displays the progress and results of the output process and notifies the user.
[0087] The photographing device 120 is, for example, a microscope device having a photographing function. The photographing device 120 photographs a sample such as cells cultured on a well plate to generate image data, and sends the image data to the interface 114 of the image processing device 110. When the photographing device 120 is set for the purpose of photographing the test image It, it is sufficient to have a bright field photographing function. On the other hand, when the photographing device 120 is set for the purpose of collecting the first training image I1 and the second training image I2, it is sufficient to have a bright field photographing function and a fluorescence photographing function. It is desirable to be able to perform bright field photographing and fluorescence photographing in the same field of view for one sample.
[0088] In addition, the structure of the image processing system is not limited to the above content, for example, the image processing device 110 and the imaging device 120 may also be configured as a whole. In addition, for example, in the case where a library of image data captured in the past can be used (for example, by receiving it through the interface 114), the image processing system 100 may not have a photographing function.
[0089] Next, an embodiment of the image processing process constructed as described above is described. The inventor of the present invention has conducted various evaluation experiments to verify the effectiveness of the image processing process of this embodiment. Thus, it is concluded that the process is effective for automatically measuring the position and number of cells based on images. A part of the results is shown below.
[0090] Figure 8 : is a diagram showing an example of the measurement result of the number of cells of the present embodiment. The inventors of the present invention used Hoechst33342 (hereinafter referred to as "Hoechst") and PI, which are markers for staining cell nuclei, to be introduced into HeLa cells, and used samples with various concentrations of a proteasome inhibitor (MG132) used as a drug to perform an experiment to count the number of live cells. Among them, Hoechst makes the nucleus of the cell emit blue light in the fluorescent image, regardless of the life or death of the cell, while PI makes the nucleus of the dead cell emit red light. Thus, the number of live cells is calculated by subtracting the number of cell nuclei counted in the red (R) image (the total number of live cells and dead cells) from the number of cell nuclei counted in the blue (B) image (the total number of live cells and dead cells).
[0091] Figure 8The solid line in the figure shows an example of the result of using the image obtained by bright field photography of the sample as the test image It, and using the first learning model 10 and the second learning model 20 to determine the position of the cell nucleus and count the number thereof. In addition, the dotted line shows an example of the result of directly inputting the image obtained by fluorescence photography of the same shooting range of the same sample into the second learning model 20 to determine the position of the cell nucleus and count the number thereof. It can be seen that the two results are roughly consistent at each drug concentration. The same comparison was performed on multiple samples, but the same result was obtained regardless of which sample was used.
[0092] It can be seen from this that the output image obtained by inputting the bright field image (i.e., the test image It) into the first learning model 10, i.e., the image with virtual markings (i.e., the intermediate image Im), has a high similarity with the marked image (fluorescence image) taken by actually introducing the markings.
[0093] Fig. 9A as well as Fig. 9B : is a graph comparing the cell count results of this embodiment with the ATP analysis method. Figure 8 same, Fig. 9A An example of the results of evaluating the number of living cells in samples treated with drugs at various concentrations using a known ATP analysis method is shown. Fig. 9B An example of the result of counting the number of living cells of a sample into which Hoechst and PI were introduced by this embodiment is shown. In these figures, the solid line indicates that the number of cells seeded into one well is 5000, and the dotted line indicates that the number of cells seeded into one well is 2000. In addition, Fig. 9B The result is a counting result in an image obtained by photographing a part of a region in one well, and therefore, the counting result does not represent the total number of cells in the well.
[0094] When the median lethal concentration (LC50) generally used as an effect index of a drug is obtained based on these results, approximately the same result is obtained by any method. The same comparison is performed on multiple samples, but the same result is obtained on any sample. Thus, it is shown that the counting result of the number of cells in the present embodiment is very consistent with the actual situation, and the evaluation of the efficacy based on the counting result obtains the same result as the ATP analysis method. In addition, the same evaluation is also performed on HepG2 cells, and it can be confirmed that the value of the LC50 obtained according to the counting result of the present embodiment is very consistent with the value of the LC50 obtained by the ATP analysis method.
[0095] That is, the cell position detection and the cell number counting based on the detection in the present embodiment can be used as a quantitative method equivalent to the ATP analysis method widely used as a standard quantitative method. The ATP analysis method is a method of invading cells such as lysing cells, and the method in the present embodiment can quantify cells based on a bright field image without using a label, etc., and is a non-invasive method, which is advantageous.
[0096] Fig.10 as well as Fig.11 is a diagram showing an example of an image of a triple-stained sample. Fig.10 This is an example of an image using HepG2 cells as a sample. Fig.11 These images are used to illustrate that the virtual fluorescence image (ie, the intermediate image Im) generated by the first learning model 10 of the present embodiment has a high similarity with the real fluorescence image.
[0097] Calcein, Hoechst, and PI are introduced into each sample as fluorescent markers. When the color image obtained by photographing the sample is decomposed into the three primary colors of RGB, the fluorescence of calcein appears as a green (G) image, the fluorescence of Hoechst appears as a red (R) image, and the fluorescence of PI appears as a blue (B) image. In the full-color image that merges the three monochrome images, the cytoplasm of living cells appears in green and its nucleus appears in blue. On the other hand, the nuclei of dead cells stained by both Hoechst and PI appear as a color close to pink in the color image due to the overlap of red and blue. Therefore, the life and death of cells can be identified based on the color of the nucleus.
[0098] By decomposing the full-color image obtained by fluorescently photographing the sample into the three primary colors of RGB, and inputting image data into the first learning model 10 established in advance based on each color, an intermediate image of each RGB color is obtained. By synthesizing (merging) these images, a full-color virtual fluorescence image is obtained. The virtual fluorescence image obtained in this way is compared with a real fluorescence image obtained by fluorescently photographing the same sample. In addition, such a full-color virtual fluorescence image can also be obtained by inputting image data into the first learning model 10 established using the full-color image as a training image.
[0099] When the images shown in these figures are observed in the horizontal arrangement, the images on the left end are bright field images that become input images to the first learning model 10. In addition, the image in the center is an intermediate image Im, which is a generated image obtained by inputting the image on the left end to the first learning model 10. In addition, the image on the right end is a fluorescent image obtained by actually taking fluorescent images of the same sample.
[0100] The ideal operation of the first learning model 10 used in the image processing process of this embodiment is to output an image identical to the color image on the right end when the bright field image on the left end is input. In this sense, the color image on the right end can be said to be the target image that the first learning model 10 should output.
[0101] In these figures, the images in the upper two rows are examples of samples without drug injection. More specifically, column A corresponds to an example of images used as training data in deep learning, and column B corresponds to images not used in training data. In addition, columns C and D are examples of samples with drug injection. Column C corresponds to an example of images used as training data in deep learning, and column D corresponds to images not used in training data.
[0102] exist Fig.10 , Fig.11 In any of the figures in FIG. 1 , in each column, when the generated image (virtual fluorescence image) in the center, which is generated and outputted based on the result of learning by the first learning model 10 provided with the input image, is compared with the image actually obtained by fluorescently photographing the sample, that is, the target image at the right end, the position of the luminous point indicating the position of the cell nucleus and its size, brightness, etc. are very consistent. This is true not only for the input image used for learning as training data, but also for the input image that is not used for learning, that is, unknown to the first learning model 10.
[0103] In addition, although the color of the image is not shown in the figure, the color between the generated image and the target image is very consistent. Specifically, in the images of columns A and B corresponding to the samples without the addition of the drug, the configuration of the blue image representing the nucleus of the living cell and the green image representing the cytoplasm is very consistent between the generated image and the target image. In addition, in the images of columns C and D corresponding to the samples with the addition of the drug, the number of cells that died due to the action of the drug increased, and corresponding to this, in addition to the blue and green images, there are also pink images. These configurations are also very consistent between the images.
[0104] Therefore, it can be said that the virtual fluorescence image output as the intermediate image Im in the image processing process of this embodiment can be fully used as a substitute for the fluorescence image obtained by actual fluorescence photography. In addition, the number of cell nuclei is counted by inputting the image used for training data and the generated image obtained by using the image as the input image to the second learning model 20, and these counting results are also very consistent.
[0105] In addition, the accuracy of the image processing process of the present embodiment including the output of the second learning model 20 is shown by the following experimental results. In the experiment, the number of cell nuclei obtained by executing the image processing process of the present embodiment using a bright field image as an input image and the number of cell nuclei obtained by providing a fluorescent image obtained by actual fluorescence photography instead of the intermediate image Im to the second learning model 20 were compared.
[0106] Specifically, a sample in which calcein as a marker is introduced into HepG2 cells is prepared, and its bright field image and fluorescence image are taken. Then, based on the result when the fluorescence image is input to the second learning model 20, the number of cell nuclei (i.e., the number of cells) is counted. The number of cells at this time is referred to as "the number from the correct image". In addition, the intermediate image Im obtained by inputting the bright field image to the first learning model 10 is input to the second learning model 20, and the number of cells is counted based on the result. The number of cells at this time is referred to as "the number from the generated image".
[0107] Fig.12 This is a graph showing an example of the experimental results. As shown in the figure, the number of images from the correct solution and the number of images from the generated image show a high degree of consistency, and when the least squares method is used for linear approximation, it becomes y = 1.0196x-3.1554
[0108] Coefficient of determination R 2 This shows that the combination of the label generation model (first learning model 10) and the localization model (second learning model 20) of the present embodiment is suitable for counting the number of cells from bright field images.
[0109] The image processing method of the present embodiment described above can be used, for example, in a drug sensitivity test method for studying the effect of a drug on cells. Next, an example of a drug sensitivity test method including the above-mentioned image processing process will be described.
[0110] Fig.13 This is a flow chart showing a drug sensitivity test method using the image processing method of this embodiment. Initially, cells to be evaluated are cultured (step S401), and a sample is prepared by adding a drug at a predetermined concentration (step S402). At a predetermined timing, such as when a certain time has passed since the drug was added, the sample is photographed in a bright field (step S403).
[0111] The above-mentioned image processing is performed on the bright field photography thus obtained. Specifically, the captured bright field image is used as the test image It, and the above-mentioned image processing process is performed on the living cells as the detected part (step S404), and the number of living cells is counted (step S405). The effect of the drug is determined based on the counting result, for example, by calculating the value of the above-mentioned LC50, etc. (step S406), and the determination result is output to the display unit 115, for example (step S407). In the case of further continuous observation and regular evaluation (step S408), return to step S403, and repeatedly perform the photography and evaluation of the sample. In this embodiment, it is possible to evaluate only by bright field photography without processing the cells, so such continuous evaluation can be performed.
[0112] As described above, in this embodiment, a first learning model 10 and a second learning model 20 are prepared in advance, wherein the first learning model 10 is established by deep learning using a combination of a fluorescence image as an example of a pre-collected labeled image and a corresponding bright field image as an example of an unlabeled image as training data, and the second learning model 20 is established by deep learning using a combination of a fluorescence image and information representing the position of a detected part in the fluorescence image as training data.
[0113] When a bright field image as a test image It is input to the first learning model 10, the first learning model 10 outputs an image in which the marker is virtually annotated on the test image It based on the learning result. That is, the first learning model 10 can produce the same image as the case where the marker is present from the bright field image without the marker. Therefore, there is no need to introduce the marker in advance, and the labeled image can be obtained in a manner that does not invade the cells. Therefore, it is also possible to observe the changes over time of a sample containing cells. In addition, since there is no need for operations and reagents for introducing the marker, it is possible to reduce the processing cost.
[0114] The second learning model 20 learns the relationship between the labeled image and the position of the detected part therein. Therefore, the second learning model 20 determines the position of the pre-selected detected part based on the image with virtual labels actually taken or output by the first learning model 10 and outputs it. There are cases where multiple cells are in contact with each other or appear to be in contact in appearance due to overlapping in the depth direction in the image. By including such examples in learning, these cells can be separated and processed. Therefore, the position and number of the detected parts in the image can be accurately measured based on the output of the second learning model 20.
[0115] As a result, in the above-mentioned embodiment, the position of the detected part in the bright field image can be determined with high accuracy by inputting the unknown bright field image, that is, the test image It, into the first learning model 10 for processing, and inputting the result into the second learning model 20 for processing. And, the detection result is suitable for the purpose of automatically measuring the position and number of the detected parts.
[0116] As described above, in the above-mentioned embodiment, the first training image I1, the second training image I2, and the third training image I3 correspond to the “first image”, “second image”, and “third image” of the present invention, respectively. Figure 7 In the image processing device 110, the interface 114 that receives image data from the outside functions as the "image acquisition unit" of the present invention. In addition, the CPU 111, especially the image processing unit 112, functions as the "image processing unit" of the present invention.
[0117] In addition, the present invention is not limited to the above-mentioned embodiment, and various changes other than the above-mentioned embodiment can be made as long as they do not deviate from the purpose. For example, in the above-mentioned embodiment, the test image It and the second training image I2 are bright field images. However, these can also be images obtained by other non-invasive imaging methods, such as phase difference images. In addition, in the above-mentioned embodiment, the first training image I1 and the third training image I3 are fluorescent images. However, these can also be other labeled images, for example, images marked with dyes that develop color under visible light illumination.
[0118] In addition, the image processing device 110 of the above-mentioned embodiment is a structure that receives image data provided from an external device via an interface 114, such as image data obtained by shooting by the shooting device 120. However, as described above, the image processing device itself may also have a shooting function. That is, the image processing system 100 as a whole can also be regarded as the "image processing device" of the present invention. In this case, the shooting device 120 that performs the shooting function functions as the "image acquisition unit" of the present invention. In addition, in the case where there is a library of images shot in the past, training data can also be produced based on the image library. In this case, the shooting function for the purpose of acquiring training images can be omitted.
[0119] In addition, for example, the device that performs the processing for establishing the first learning model and the second learning model and the device that processes the test image It using the results thereof may also be different. For example, it may also be configured such that the first learning model 10 and the second learning model 20 established by performing deep learning in a certain image processing device are transplanted to other image processing devices, and the transplanted devices process the test image It. If the establishment of the learning model is performed using a device with high computing power, such as a supercomputer, a high-precision learning model can be established using a large amount of data. On the other hand, in a device that processes the test image It using the established learning model, it can be completed using only relatively simple calculations, and therefore, it is not necessary to have high computing power. Thus, for example, as described below, it is possible to perform processing using a relatively simple computing device built into the device.
[0120] Fig.14 is a diagram showing another embodiment of the present invention. More specifically, Fig.14 2 is a block diagram of a microscope device 200 of another embodiment of the image processing device of the present invention. The microscope device 200 includes a microscopic imaging unit 210 including an illumination light source 211, an imaging optical system 212, and an imaging element 213, a controller 220, and a video interface (IF) 230. That is, this embodiment is a microscope device with a video output function. The microscopic imaging unit 210 only needs to have a function of performing bright field imaging or phase difference imaging of a sample, and does not need to have a fluorescence imaging function.
[0121] The controller 220 has a function as a control body that controls each device unit to execute a photographing operation. In addition, the controller 220 performs appropriate image processing on the image photographed by the photographing unit 210 and outputs the image to the video interface 230. For this purpose, the controller 220 is provided with an image processing processor 221.
[0122] The image processing processor 221 is a component that combines the first learning model 10 and the second learning model 20 that have been learned as fixed or replaceable data into dedicated hardware. For example, the image processing processor 221 can be configured by ASIC (Application Specific Integrated Circuit), PLA (Programmable Logic Array), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), etc. The image processing processor 221 has a function of outputting image data corresponding to the intermediate image Im generated by the first learning model 10 and the output image Io generated by the second learning model 20. It may also have a function of superimposing and synthesizing various images.
[0123] The image data obtained by the imaging of the imaging unit 210 or the image data processed by the image processing processor 221 is output to the external display device 300 via the video interface 230. The display device 300 displays and outputs an image corresponding to the image data provided from the microscope device 200. In this embodiment, the imaging unit 210 functions as the "image acquisition unit" of the present invention, and the image processing processor 221 functions as the "image processing unit" of the present invention. Instead of outputting to the display device 300 or in addition, the processing result may be output to a suitable external device 400 via the interface 230. As the external device 400, an external storage device for storing the processing result, a computer device for receiving the processing result and performing further image processing and analysis, etc. can be applied.
[0124] According to such a structure, the image captured by the shooting unit 210 can be processed in sequence by the image processing processor 221 and displayed on the display device 300. Here, if the intermediate image Im output by the image processing processor 221 is displayed on the display device 300, the microscope device 200 that does not have a fluorescence shooting function can be used as a virtual fluorescence microscope. That is, the microscope device 200 has the function of outputting a fluorescence image. If the established learning model is instrumentalized in this way, it can be given a completely new function, for example, by combining it with an existing microscope device. By using dedicated hardware dedicated to image processing, the time required for processing can be shortened. For example, if the processing of an image can be completed in about 0.1 seconds or less, it can essentially become real-time image processing.
[0125] For example, it is possible to perform time-lapse photography in which photography is intermittently performed at time intervals equal to the time required for image processing, perform image processing each time, and display the intermediate image Im on the display device 300. In this way, the sample can be observed in real time through a virtual fluorescent image, similar to a fluorescence microscope. In addition, if the output image Io output by the second learning model 20 or the counting result based on the output image Io is displayed on the display device 300, the state of the sample that changes from moment to moment can be quantitatively grasped.
[0126] For example, a learning model can be established by using a structure that appears only at a specific period in the activity of a cell (for example, a chromosome that appears temporarily during the process of cell division) as a detected part. In this way, an output suitable for observing cell division can be obtained. That is, if the intermediate image Im is displayed on the display device 300, when the detected part appears in the activity of the cell, the characteristic mark corresponding to it appears in the display image, that is, the virtual fluorescent image. Therefore, the appearance of the part can be grasped immediately. In addition, the cell cycle can also be quantitatively tracked based on the changes in the counting results of the detected part that appears only at a special period of the cell cycle.
[0127] Such applications become possible if it is possible to use a marker that selectively stains a site that temporarily appears at each stage of cell activity, and if it is possible to collect a sufficient number of images in which the marker appears as training images. That is, by using such a labeled image and its corresponding bright field image or phase difference image as training data to establish a learning model, and inputting the bright field image or phase difference image of a sample without a marker into the learning model, it is possible to obtain various information related to cell characteristics, such as cell life and death, cell cycle, etc.
[0128] In addition, for example, if a learning model is established using an image of a sample with a label indicating that undifferentiated cells and differentiated cells have different appearances as a training image, it is possible to identify undifferentiated cells and differentiated cells from an image of a sample without the label, and to obtain quantitative information suitable for observing the differentiation process of cells by counting their numbers separately.
[0129] As the specific embodiments are illustrated and described above, in the present invention, for example, at least a portion of the plurality of first images may also be used as at least a portion of the plurality of third images. Or, for example, the plurality of first images and the plurality of third images may also be different from each other. Moreover, for example, the output image when the second image is input to the first learning model may be the third image. In addition, third images obtained by different methods may be mixed. In this way, images obtained by various methods can be used as the third image, which is suitable for collecting multiple examples to improve the learning effect.
[0130] In addition, for example, the test image and the second image may be a bright field image or a phase difference image. As the test image and the second image, an image without a marker can be used, and therefore, an image obtained by photographing a sample without a marker can also be used. If a bright field image or a phase difference image is used as such an image, high affinity can be obtained even by visual observation.
[0131] In addition, for example, the first image may be a fluorescent image obtained by photographing fluorescently labeled cells under illumination with excitation light. In fluorescent labeling technology, many methods have been developed for selectively displaying labels on specific parts of cells according to the purpose. By applying such established methods, the target detection part can be reliably detected.
[0132] Here, the fluorescent label may include a label showing different appearances in live cells and dead cells. According to such a configuration, live cells and dead cells can be identified from images without introducing a label.
[0133] In addition, for example, the image processing method of the present invention may also be a structure that counts and outputs the number of detected parts based on the result image. As described above, according to the present invention, the positions of representative points of the detected parts are detected, so that, for example, even if a plurality of detected parts having width overlap in the image, these detected parts can be detected separately. Thus, by counting the number of detected representative points, the number of detected parts can be automatically counted.
[0134] The present invention has been described above according to specific embodiments, but the description is not intended to be interpreted in a limiting sense. With reference to the description of the present invention, various modifications of the disclosed embodiments will be apparent to those skilled in the art, as with other embodiments of the present invention. Therefore, the appended claims include such modifications or embodiments within the scope of the true scope of the present invention.
[0135] Industrial Applicability
[0136] Since the present invention can determine the position of a specific part in an image containing cells in a non-invasive manner, it is suitable for observing the changes of a sample containing cells over time. Therefore, it is particularly suitable for applications such as drug sensitivity tests for studying the effects of drugs on cells and screening of innovative drugs such as anticancer drugs based on the tests.
[0137] Description of reference numerals:
[0138] 10. First Learning Model
[0139] 20 Second Learning Model
[0140] 100 Image Processing System
[0141] 110 Image processing device
[0142] 111 CPU
[0143] 112 Image Processing Department
[0144] 114 Interface (image acquisition unit)
[0145] 120 Photographic device (image acquisition unit)
[0146] 200 Microscope device (image processing device)
[0147] 210 Photographing unit (image acquisition unit)
[0148] 221 Image processing processor (image processing unit)
[0149] I1 First training image (first image)
[0150] I2 Second training image (second image)
[0151] I3 The third training image (the third image)
[0152] Ip location information image
[0153] Im Intermediate Image
[0154] Io result image
[0155] It test image.
Claims
1. An image processing method for detecting the position of a specific detected part based on a test image obtained by photographing a cell, comprising: A step of inputting the test image in which the mark corresponding to the detected part is not visible to a first learning model, and generating an image corresponding to the test image and having the virtual mark as an intermediate image; The step of inputting the intermediate image to a second learning model, generating an image in which the position of the detected part is indicated by its representative point, and outputting the image as a result image, The first learning model is established by performing deep learning by using training data in which a first image in which the mark appears and a second image in which the mark does not appear are associated with each other, so that the second image corresponds to an input and the first image corresponds to an output, wherein the first image and the second image are images taken including cells identical to each other, The second learning model is established by using training data that associates an image taken of a cell, i.e., a third image in which the mark appears, with information representing the position of the representative point contained in the third image, so that the third image corresponds to input and the position of the representative point corresponds to output to perform deep learning.
2. The image processing method according to claim 1, wherein: At least a portion of the plurality of first images is used as at least a portion of the plurality of third images.
3. The image processing method according to claim 1, wherein: The plurality of first images and the plurality of third images are different from each other.
4. The image processing method according to claim 1, wherein: The output image after the second image is input into the first learning model is used as the third image.
5. The image processing method according to any one of claims 1 to 4, wherein: The test image and the second image are bright field images or phase difference images.
6. The image processing method according to any one of claims 1 to 4, wherein: The first image is a fluorescent image obtained by photographing cells marked with fluorescence under illumination of excitation light.
7. The image processing method according to claim 6, wherein: The fluorescent markers include markers that show different appearance patterns in living cells and dead cells.
8. The image processing method according to any one of claims 1 to 4, wherein: The number of the detected parts is counted based on the result image and outputted.
9. A drug sensitivity test method comprising: a step of acquiring an image obtained by photographing a sample in which a drug to be evaluated is administered to cultured cells; A step of using the acquired image as the test image and counting the number of living cells in the detection site using the image processing method according to claim 8; A step of determining the sensitivity of the cells to the drug based on the counting result.
10. An image processing device for detecting the position of a specific detected part based on a test image obtained by photographing cells, comprising: An image acquisition unit, which acquires a bright field image or a phase difference image in which a mark corresponding to the detected part is not visible as the test image; as well as an image processing unit that performs image processing, inputs the test image to the first learning model, generates an image corresponding to the test image and having the virtual marker as an intermediate image, and inputs the intermediate image to the second learning model, generates and outputs an image in which the position of the detected part is indicated by its representative point as a result image, The first learning model is established by performing deep learning by using training data in which a first image in which the mark appears and a second image in which the mark does not appear are associated with each other, so that the second image corresponds to an input and the first image corresponds to an output, wherein the first image and the second image are images taken including cells identical to each other, The second learning model is established by using training data that associates an image taken of a cell, i.e., a third image in which the mark appears, with information representing the position of the representative point contained in the third image, so that the third image corresponds to input and the position of the representative point corresponds to output to perform deep learning.
