Image processing method and classification model construction method

Abstract

To construct a classification model capable of accurately extracting a cell region in an image from any of two types of bright field images having different contrast states.SOLUTION: An image processing method includes the steps of: obtaining a ground truth image teaching a cell region occupied by a cell in an original image for each of a plurality of original images obtained by bright-field imaging of the cell; generating a reverse image by reversing luminance of the original image at least for the cell region based on each original image; constructing a classification model by performing machine learning using a set of the original image and the ground truth image corresponding to the original image and a set of the reverse image and the ground truth image corresponding to the original image as a basis of the reverse image, respectively, as training data; and inputting a bright-field image of two-dimensionally cultured cells, as a test image, to the classification model, and acquiring an output image thereof.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing method for identifying the area occupied by a cell from an image of the cell, and a method for constructing a classification model for executing the method using a machine learning model. [Background technology]

[0002] In medical and biological science experiments, cells to be observed are captured using a CCD camera or the like, converted into digital image data, and then various image processing techniques are applied to the image data for observation and analysis. For example, by collecting a large number of images of cells as training images and providing the regions occupied by cells in these images along with a ground truth image to a classification model for machine learning, it is possible to construct a classification model that can automatically extract the regions occupied by cells from newly provided test images.

[0003] When a cell image is a bright-field image, e.g., captured using an optical microscope, the cell is nearly transparent, so the cell image is not necessarily clear in the in-focus image. In fact, an image captured slightly out of focus may provide better cell visibility. For example, when the cells being imaged are two-dimensionally cultured (plate cultured), it is known that the contrast between the edge and interior of the cell is reversed depending on the focal position during imaging. Specifically, there are cases where the edge of the cell is bright, i.e., highly luminous, and the interior is darker, i.e., less luminous (hereinafter referred to as the "first contrast state"), and cases where the edge is dark and the interior is brighter (hereinafter referred to as the "second contrast state") (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-163248 Summary of the Invention [Problem to be solved by the invention]

[0005] Both of these two types of images with different contrasts are suitable for observing cells, and which image is acquired is determined as appropriate mainly depending on the photographer's preference, the environment during imaging, etc. Furthermore, depending on the state of the sample during imaging, the two types of contrast conditions described above may be mixed in the image.

[0006] It is conceivable that a large number of images obtained in this manner are collected and used as training images for constructing a classification model for automatically extracting the above-mentioned cellular region. In this case, the contrast state of both the training images and the test images to be subjected to the automatic extraction process may be either the first contrast state or the second contrast state.

[0007] If the training images used in machine learning contain many images captured under the same contrast conditions as the test images, automatic extraction using the classification model is expected to function with high accuracy, but if this is not the case, the extraction accuracy will be lower. Furthermore, it is more preferable to obtain the same extraction accuracy regardless of the contrast conditions under which the test images are captured.

[0008] For these reasons, it is desirable to collect teacher images from a balanced collection of images taken under two contrast conditions. This can be taken into consideration when taking new images to prepare teacher images, but the amount of work required to capture multiple specimens while achieving two focus states and collecting teacher images is enormous. While the amount of work could be significantly reduced if previously captured images could be used as a library, it is not always possible to prepare a library that contains a balanced collection of images under the two contrast conditions.

[0009] This invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that can construct a classification model that can accurately extract cellular regions in an image from either of two types of bright-field images with different contrast conditions. [Means for solving the problem]

[0010] In order to achieve the above object, one aspect of the present invention includes a step of acquiring, for a plurality of original images of cells captured in bright field, a ground truth image indicating the cell region occupied by the cell in the original image; a step of creating an inverted image by inverting the brightness of at least the cell region based on each of the original images; and a step of creating a set of the original image and the corresponding ground truth image, and a set of the inverted image and the ground truth image corresponding to the original image on which the inverted image was based. include Teacher data Based on and performing machine learning to build a classification model.

[0011] Another aspect of the present invention is a method for constructing a classification model for identifying an area occupied by a cell from a bright-field image of a two-dimensionally cultured cell, and in order to achieve the above object, the method includes the steps of: obtaining, for a plurality of original images of cells obtained by bright-field imaging, a ground truth image that indicates the cell area occupied by the cell in the original image; creating an inverted image based on each of the original images by inverting the brightness of at least the cell area; and creating a set of the original image and the corresponding ground truth image, and a set of the inverted image and the ground truth image that corresponds to the original image on which the inverted image was based. include Teacher data Based on and a step of performing machine learning to build a classification model.

[0012] With the invention configured in this way, it is possible to construct a classification model that can accurately extract cellular regions in images, regardless of which of the two contrast states each of the original images collected to be used as training images has. Moreover, it is possible to obtain good extraction results regardless of which of the two contrast states the input image input to the classification model as the target for cellular region extraction has. The reasons for this are as follows.

[0013] As mentioned above, there are two contrast states in a bright-field image of a cell. Specifically, in the first contrast state, the edge (contour) of the cell is bright and the interior is darker. On the other hand, in the second contrast state, the edge is dark and the interior is brighter. Thus, the relationship of image contrast between the edge and interior of the cell is symmetric between the two contrast states.

[0014] Therefore, it is expected that by inverting the brightness of an image, it is possible to convert between two contrast states and create an inverted image that simulates one contrast state from an original image of the other contrast state. Although such a simple inverted image is not necessarily suitable for observing cells, the inventors' experiments have confirmed that it is sufficiently effective as a training image for building a classification model.

[0015] The original image is acquired to construct a classification model for automatically extracting cell regions from bright-field images, and is paired with a ground truth image that indicates the cell regions in the original image. In principle, the areas occupied by cells are identical between the original image and the inverted image. In other words, the ground truth image also indicates the cell regions in the inverted image.

[0016] Therefore, the set of the original image and the correct image, and the set of the inverted image and the correct image, can be treated as training data corresponding to the same cells but with different contrast conditions. This enriches the training data and allows for the collection of examples in the two contrast conditions without bias. By performing machine learning using the training data collected in this way, the constructed classification model has the generalization ability to handle input images in either contrast condition, making it possible to accurately extract cell regions from any input image.

[0017] Note that this contrast inversion is a phenomenon occurring within the cellular region. Therefore, when creating an inverted image, it is sufficient to invert the brightness at least in the cellular region of the original image. Since the cellular region in the original image is designated as the correct image, it is also possible to selectively invert only the cellular region of the original image.

[0018] Furthermore, the above method does not distinguish between the contrast conditions under which the original image and the input image were captured. Therefore, when collecting training images and capturing input images, it is not necessary to specify the contrast conditions under which the images were captured. Furthermore, even images containing a mixture of two contrast conditions can be used without any problems as original images for constructing a classification model and as input images for the constructed classification model. [Effects of the Invention]

[0019] According to this invention, by inverting at least the region designated as a cellular region in an original image acquired as a training image and performing machine learning using the original image and the inverted image as training images, it is possible to construct a classification model that can accurately extract cellular regions from an image from either of two bright-field images with different contrast conditions. Furthermore, the classification model constructed in this way can accurately extract cellular regions from input images captured under either contrast condition. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram showing a schematic configuration of an embodiment of an imaging device capable of executing an image processing method according to the present invention. [Figure 2] 4 is a flowchart showing image processing in this embodiment. [Figure 3] FIG. 1 shows an example of a bright-field image of cells cultured on a plate. [Figure 4] 1 is a flowchart illustrating a method for constructing a classification model according to the present embodiment. [Figure 5] 10 is a flowchart showing a first example of a reverse image creation process. [Figure 6] 10 is a flowchart showing a second example of the reverse image creation process. [Figure 7] 10 is a flowchart showing a fourth example of the reverse image creation process. [Figure 8] FIG. 1 shows the case of an original image and an inverted image. [Figure 9] FIG. 10 is a diagram showing an example of a cell region extraction result according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] 1 is a diagram showing the schematic configuration of one embodiment of an imaging device capable of executing an image processing method according to the present invention. This imaging device 1 is a device for imaging a sample S, such as cells, held in a flat, open-topped sample container 10 called a dish. A predetermined amount of liquid, serving as a culture medium M, is poured into the sample container 10, and the sample S, such as cells, cultured in this liquid under predetermined culture conditions becomes the object of imaging by this imaging device 1. The culture medium may contain an appropriate reagent, or may be one that gels after being poured into the sample container 10 in liquid form.

[0022] In addition to the above, the imaging subject may be, for example, a tissue slice or a pathological specimen carried on an appropriate carrier. Also, the sample container may be a so-called well plate having a plurality of depressions called wells, and the imaging subject may be a sample carried in each well.

[0023] The imaging device 1 includes a holder 11 that holds a sample container 10, an illumination unit 12 arranged above the holder 11, an imaging unit 13 arranged below the holder 11, and a control unit 14 having a CPU 141 that controls the operation of each of these units. The holder 11 abuts against the peripheral portion of the bottom surface of the sample container 10 and holds the sample container 10 in a substantially horizontal position.

[0024] The illumination unit 12 emits illumination light toward the sample container 10 held by the holder 11. A white LED (Light Emitting Diode), for example, can be used as a light source for the illumination light. A combination of the light source and an appropriate illumination optical system is used as the illumination unit 12. The illumination unit 12 illuminates the object to be imaged in the sample container 10 from above.

[0025] An imaging unit 13 is provided below the sample container 10 held by the holder 11. The imaging unit 13 has an imaging optical system disposed directly below the sample container 10, with the optical axis of the imaging optical system oriented vertically. Figure 1 is a side view, and the up-down direction in the figure represents the vertical direction.

[0026] The imaging unit 13 images the imaging target in the sample container 10. Specifically, light emitted from the illumination unit 12 and incident on the liquid from above the sample container 10 illuminates the imaging target, and light transmitted downward from the bottom of the sample container 10 is incident on the light receiving surface of the imaging element 132 via an imaging optical system including an objective lens 131 of the imaging unit 13. An image of the imaging target formed on the light receiving surface of the imaging element 132 by the imaging optical system is captured by the imaging element 132. The imaging element 132 is an area image sensor having a two-dimensional light receiving surface, and for example, a CCD sensor or a CMOS sensor can be used.

[0027] The imaging unit 13 can be moved in the horizontal and vertical directions by a mechanical control unit 146 provided in the control unit 14. Specifically, the mechanical control unit 146 operates the drive mechanism 15 based on a control command from the CPU 141 to move the imaging unit 13 in the horizontal direction, thereby moving the imaging unit 13 in the horizontal direction relative to the sample container 10. Furthermore, focus adjustment is performed by moving the imaging unit 13 in the vertical direction.

[0028] When the driving mechanism 15 moves the imaging unit 13 in the horizontal direction, it moves the illumination unit 12 integrally with the imaging unit 13 as shown by the dotted arrow in the figure. That is, the illumination unit 12 is disposed so that its optical center substantially coincides with the optical axis of the imaging unit 13, and moves in conjunction with the imaging unit 13 when the imaging unit 13 moves in the horizontal direction. As a result, even when the imaging unit 13 moves relative to the sample container 10, the optical center of the illumination unit 12 is always positioned on the optical axis of the imaging unit 13, and the illumination conditions for the imaging object can be kept constant at any position, thereby maintaining good imaging conditions.

[0029] An image signal output from the image sensor 132 of the imaging unit 13 is sent to the control unit 14. That is, the image signal is input to an AD converter (A / D) 143 provided in the control unit 14 and converted into digital image data. The CPU 141 performs appropriate image processing based on the received image data.

[0030] The control unit 14 further has a memory 144 for temporarily storing image data and data being calculated, and a storage 145 for storing and saving programs to be executed by the CPU 141 and data generated by the CPU 141. The memory 144 can be accessed at high speed by the CPU 144, but has a smaller storage capacity than the storage 145. The storage 145 is, for example, a hard disk drive (HDD), which has a large storage capacity but a slower access speed than the memory 144. These are used according to the purpose. The CPU 141 performs various calculation processes, which will be described later, by calling and executing control programs stored in the storage 145.

[0031] Additionally, the control unit 14 is provided with an interface (IF) unit 142. The interface unit 142 has a user interface function of accepting operation inputs from the user and presenting information such as processing results to the user, as well as a function of exchanging data with an external device connected via a communication line. To realize the user interface function, the interface unit 142 is connected to an input accepting unit 147 that accepts operation inputs from the user and a display unit 148 that displays messages to the user, processing results, and the like.

[0032] Next, an image processing method using the imaging device 1 configured as described above will be described. This image processing method corresponds to one embodiment of the image processing method according to the present invention. The purpose of this processing is to automatically extract the area occupied by the cells from a bright-field image captured by the imaging device 1 using cells C as sample S, which are plate-cultured (two-dimensionally cultured) along the bottom surface of the container 10.

[0033] 2 is a flowchart showing image processing in this embodiment. This processing is realized by the CPU 141 of the imaging device 1 executing a control program prepared in advance and causing each component of the device to perform a predetermined operation. First, a bright-field image of the sample S is acquired (step S101). Specifically, a sample container 10 in which cells as the sample S are two-dimensionally cultured is set in the holder 11 of the imaging device 1, and the imaging unit 13 of the imaging device 1 captures an image while scanning the sample container 10, thereby acquiring a bright-field image of the sample S.

[0034] The acquired bright-field image is input to a classification model previously constructed by machine learning (step S102). As will be described in detail later, this learning model has the function of distinguishing, as a result of machine learning, the area occupied by cells (hereinafter referred to as the "cell area") from the other area (hereinafter referred to as the "background area") in the bright-field image input as a test image. For example, it outputs a mask image for dividing the test image into the cell area and the background area.

[0035] Based on the output image of the classification model, the original bright-field image is segmented into a cell region and a background region (steps S103 and S104). That is, by applying the mask image output by the classification model to the original bright-field image, only the region occupied by cells is extracted from the bright-field image. This separates the cell region from the background region.

[0036] A method for constructing a classification model for implementing the above image processing will now be described. The use of such classification models is widespread for the purpose of extracting regions with specific appearance characteristics from a test image. Various machine learning algorithms are known for constructing classification models suitable for this purpose, and an appropriate selection from these algorithms can be used in this embodiment. Deep learning algorithms, which do not require artificially setting feature quantities used for classification, are suitable for analyzing images of cells with indefinite shapes and large individual variations. Here, a convolutional neural network is used as an example, but the present invention is not limited to this.

[0037] Building a classification model using machine learning involves collecting typical examples that serve as training images, labeling those training images (instruction input), and running machine learning using these as training data. To accomplish this, it is necessary to collect a large number of images of cells that are the same type as or have similar characteristics to the images of the cells to be classified. For example, if there is a library of previously captured images, these can be used. However, if the target of processing is bright-field images of cells cultured on a plate, the following points must be considered.

[0038] Figure 3 shows an example of bright-field images of flat-plate cultured cells. The focal position relative to the sample during imaging is slightly different between Figures 3(a) and 3(b), resulting in differences in the contrast of the cell images. Specifically, in image Ia shown in Figure 3(a), the cell outline (edge) appears white, while the interior of the cell appears darker. In other words, the cell outline in image Ia is relatively bright, while the interior of the cell is relatively dark. On the other hand, in image Ib shown in Figure 3(b), the opposite occurs: the cell outline appears dark, while the interior of the cell appears whiter. In other words, the cell outline in image Ib is relatively dark, while the interior of the cell is relatively bright. These differences are thought to be caused by flat-plate cultured cells spreading thinly along the container wall and acting like a convex lens.

[0039] As shown in FIG. 3(c), image Ia is an image obtained when imaging is performed in a state where the focal plane Fa of the objective lens 131 is slightly behind the cell C. Image Ib is an image obtained when imaging is performed in a state where the focal plane Fb of the objective lens 131 is slightly in front of the cell C. Strictly speaking, these states are outside the focusing conditions for the cell C, but both are suitable for observing the cell C. This is because, since the cell C is nearly transparent, the visibility of the cell C is actually low in an image in a completely focused state, and the image of the cell C appears clearer in a state where the focal position is slightly off, as in images Ia and Ib.

[0040] Thus, bright-field images suitable for observing two-dimensionally cultured cells can be classified into two types: an image in a first contrast state, in which the cell outlines are bright and the center is dark, as in image Ia shown in Figure 3(a), and an image in a second contrast state, in which the cell outlines are dark and the center is bright, as in image Ib shown in Figure 3(b). There is no fundamental difference in merit between these, and the type is selected based on the observer's preferences, purpose, imaging environment, etc.

[0041] The examples collected as candidate training images may contain images with these two contrast states. There may be cases where the images are biased toward one of the contrast states, or cases where both are mixed. Furthermore, the test images to be processed may also have two contrast states.

[0042] It is desirable that training images be captured under the same contrast conditions as the test images. However, especially when using a library of previously captured images, it is not always possible to collect the necessary number of such cases, and the task of distinguishing between the contrast conditions of a large number of images places a significant burden on the operator. Furthermore, if the test image is captured under a different contrast condition, sufficient classification accuracy cannot be achieved. Furthermore, depending on the cell position and the imaging environment, two contrast conditions may be mixed in a single image.

[0043] For these reasons, it is preferable that the training images used in the training data include images captured under two contrast conditions to an equal extent. However, it remains the case that it is not always possible to prepare the required number of such images.

[0044] Therefore, the inventors of the present application have focused on the characteristic that the image contrast of cells is symmetrical between the two types of images described above, and have come up with the following idea. That is, by inverting the contrast of an image collected as a training image, it is possible to create a pseudo image in one contrast state from an image in the other contrast state. Then, by using both of the images in the two contrast states thus obtained as training images, it is possible to enrich the training examples.

[0045] By using the images collected in this way under two different contrast conditions as training images, it is expected that it will be possible to construct a classification model that has equivalent classification accuracy for test images under either contrast condition, i.e., has excellent generalization performance. Furthermore, verification experiments conducted by the inventors of this application have confirmed that by using such pseudo-generated training images, a classification model with sufficient classification accuracy for test images under either contrast condition can be obtained. A specific method for constructing such a classification model will be described below.

[0046] 4 is a flowchart showing a method for constructing a classification model in this embodiment. This process can be implemented by the CPU 141 of the imaging device 1 executing a prepared control program. However, if no new imaging is required to obtain a training image, this process can be executed by a general-purpose computer device having a general hardware configuration. Because high computing power is required to execute a machine learning algorithm, it is desirable to use a computer device with higher performance than the control unit 14 provided in the imaging device 1.

[0047] First, an original image that can be used as a teacher image is acquired (step S201). The original image can be a bright-field image of cells of the same type as the sample S to be processed, but the contrast state at the time of image capture does not matter. That is, the original image can be in either the first contrast state or the second contrast state, or it can be either one alone or a mixture of the two. The ratio between the first and second contrast states is also arbitrary. It is more preferable that the cells in the original image are two-dimensionally cultured like the sample S. The original image can be newly captured by the imaging device 1, or it can be acquired from an image library that collects images captured in the past.

[0048] Next, instruction input from the user for each original image is accepted (step S202). This instruction input is an operation for labeling the area of ​​the image occupied by cells. For example, the original image can be displayed on the display unit 148, and instruction input can be accepted by user input via the input accepting unit 147.

[0049] A correct image is created based on the teaching results (step S203). The correct image is an image in which the original image is explicitly divided into a cell region occupied by cells and a background region other than the cell region. For example, an image in which the brightness of the cell region and the background region is binarized can be used. Such an image can be used as a mask image when extracting only the cell region (or background region) from the original image.

[0050] Next, an inverted image is created based on the original image (step S204). The specific processing method will be described later, but the inverted image is an image in which the image contrast of the cellular region in the original image is inverted. When the original image is an image of a cell captured under a first contrast state, the inverted image is a pseudo-representation of the image that would be obtained when the same cell is captured under a second contrast state. Through the processing up to this point, multiple pairs of original images and their corresponding correct images and inverted images are created.

[0051] Both the original image and the inverted image are used as training images, thereby enriching the training images. Specifically, one set of training data is created by pairing the original image with its corresponding ground truth image (step S205), and another set of training data is created by pairing the inverted image with its corresponding ground truth image (step S206). The original image and the inverted image are images of the same cells captured in the same field of view but under different contrast conditions. Therefore, the cell region indicated in the original image automatically indicates the cell region in the inverted image.

[0052] In this way, two training images are prepared from one original image, and the training work only needs to be performed on one of the images. Therefore, the workload of the user does not increase when enriching the training data.

[0053] Based on the teacher data thus prepared, machine learning is performed using an appropriate learning algorithm (step S207). As described above, a learning algorithm based on the principles of deep learning can be suitably applied. Furthermore, as a method for segmenting an image into regions, for example, a known semantic segmentation method can be applied.

[0054] The classification model constructed by the above process has the ability to segment a test image into a cell region and a background region for both the first and second contrast conditions. When the binarized mask image is used as the correct image, the output image of the classification model also serves as a mask image for segmenting the test image into a cell region and a background region. By applying this mask image to the test image and performing appropriate image processing, it is possible to distinguish between the cell region and the background region in the test image. Examples of image processing in this case include, but are not limited to, a process for cutting out either the cell region or the background region from the image, a process for applying different visual effects (e.g., color coding) to the cell region and the background region, and a process for emphasizing the contours of the cell region.

[0055] Next, several examples of the process for creating an inverted image from an original image (step S204 in FIG. 4) will be described with reference to FIG. 5 to FIG. 7. Four specific examples will be described here, but with the objective of creating a pseudo-teacher image from an original image, it is possible to obtain equivalent results using any of the examples.

[0056] FIG. 5 is a flowchart showing a first example of the inverted image creation process. In this example, the luminance of the entire original image is inverted (step S301). This inverts the image contrast of the cellular region, but the background region is also inverted, resulting in a density of the entire image that differs significantly from the actual one. Therefore, the average luminance of each of the original image and the inverted image is calculated (steps S302 and S303), and the luminance value of each pixel constituting the inverted image is scaled so that the two images roughly match (step S304). This creates an inverted image.

[0057] In this process, cell regions and background regions in the original image are not distinguished. Therefore, after acquiring the original image, it is possible to immediately create an inverted image without waiting for instruction input. In contrast, in each example described below, it is assumed that cell regions and background regions in the original image have been distinguished in advance. In this embodiment, such a distinction is possible through instruction input from the user, so a process based on this assumption can be adopted.

[0058] FIG. 6 is a flowchart showing a second example of the inverted image creation process. In this example, as in the first example, the luminance of the entire original image is inverted (step S401). However, luminance scaling is performed only for the background region. That is, the average luminance is calculated for the background region of each of the original image and the inverted image (steps S402 and S403), and the luminance value of each pixel constituting the background region of the inverted image is scaled so that these average luminances become approximately equal (step S404). In this way, an inverted image is created.

[0059] Inverting the brightness simply inverts the image contrast in the cell region, but causes a large change in the overall brightness in the generally uniform background region. However, in actual imaging, a slight difference in the focus position setting does not result in a large change in background density. Taking this into consideration, in this example, only the brightness of the background region after inversion is scaled. This maintains the image contrast in the cell region. Furthermore, because the brightness of the background region in the inverted image is not affected by the brightness of the cell region, the difference in brightness of the background region before and after inversion can be reduced.

[0060] Next, a third example of the inverted image creation process will be described. In this example, the brightness of only the area of ​​the original image designated as the cellular area is inverted. Therefore, the background area remains unchanged in the inverted image, and the inverted image can be used as an inverted image without scaling. Since this process simply inverts the brightness of the cellular area, a flowchart is not shown.

[0061] Figure 7 is a flowchart showing a fourth example of the inverted image creation process. In this example, as in the third example, the luminance of only the cellular region is inverted (step S501), while the luminance of the background region remains unchanged. However, the luminance of the inverted cellular region is scaled to adjust image contrast. That is, the average luminance of the cellular region in the original image and the inverted image is calculated (steps S502 and S503), and the luminance value of each pixel constituting the inverted cellular region is scaled so that the average luminances are approximately equal (step S504).

[0062] Depending on the imaging conditions such as the state of the cells and lighting, simple inversion may result in unnatural brightness and image contrast in the cell region. Scaling the inverted cell region can solve this problem.

[0063] Any of these examples may be used for the process of creating an inverted image in step S104, and these may be switched between as needed. For example, the processing method may be selected by a user operation. Note that although the scaling in each of the above examples is based on the average brightness, this is merely an example, and scaling based on the median brightness, the difference in brightness (or contrast value) between the cell region and the background region, or the like may also be performed.

[0064] Figure 8 shows examples of original and inverted images. Image Ic shown in Figure 8(a) is the original image, captured under the first contrast condition, in which the cell outlines appear bright and the interior of the cell appears darker. Image Id shown in Figure 8(b) is an example of an inverted image created from image Ic using the method described in the first example above. The cell outlines appear dark and the interior of the cell appears bright, indicating that the image contrast has been inverted. Meanwhile, the density of the background region remains almost the same as in the original image Ic. Image Id is a pseudo-representation of the image that would be obtained by capturing an image under the second contrast condition.

[0065] FIG. 9 shows an example of the results of cell region extraction using this embodiment. FIG. 9(a) is shown as a comparative example. Image Ie shows the results of extracting cell regions from a test image obtained by constructing a classification model using only images captured under the first contrast condition as training images and inputting a test image captured under the second contrast condition into the model. Multiple cells are distributed in image Ie, but the area enclosed by the dotted line was not extracted as a cell region. As shown, there is a discrepancy between the actual image and the extraction results, such as entire cells being omitted from the extraction target or partial cell regions not being extracted, indicating insufficient extraction accuracy.

[0066] On the other hand, image If shown in Figure 9(b) shows the extraction results using a classification model constructed using both the original image and an inverted image created from it as training images. The area occupied by the cell images in the image matches well with the extracted area, demonstrating improved extraction accuracy.

[0067] As described above, in the image processing of this embodiment, the classification model for analyzing bright-field images of cells cultured on a flat surface (two-dimensional culture) and extracting the cell region occupied by the cells is constructed by machine learning using both the collected original images and the inverted images created from them as training images. This configuration has the following advantages:

[0068] First, by enriching the number of training examples, the generalization performance and classification accuracy of the classification model can be improved. Specifically, regardless of whether the image of the cell to be processed is captured in the first or second contrast state, it is possible to extract the cell region from the image with high accuracy.

[0069] Second, when performing this image processing, it is not necessary to distinguish the contrast state the image was captured in. This has the following implications.

[0070] First, regardless of the contrast condition under which the training image was captured, it contributes to enriching the case studies. Therefore, the contrast conditions of the collected images may be biased, or even only images in one contrast condition may be included. Because the contrast condition at the time of capture is often selected based on the observer's preference, for example, images collected within a single research facility may be biased toward one contrast condition. Even in such a situation, the case studies can be enriched by creating a simulated image corresponding to the other contrast condition and using it as a training image together with the original image. Therefore, for example, a library of previously captured images can be used as training images.

[0071] Furthermore, there is no need to distinguish between the contrast conditions under which each of the collected images was captured. Even if images with two contrast conditions are mixed, they do not need to be distinguished when used as training images, and the technology and effort required for this is unnecessary. Therefore, the collected images can be used as training images without waste and without any effort.

[0072] Furthermore, the image to be processed can be captured in any contrast state. Therefore, the user can capture an image in a contrast state that suits their purpose or preference without worrying about the convenience of image processing. Furthermore, even when processing an image that has already been captured, good processing results can be obtained regardless of the contrast state in which the image was captured, and the user does not need to be aware of this.

[0073] Furthermore, even an image in which two contrast states are mixed in a single image can be used as the original image or the image to be processed in this embodiment. Images in which two contrast states are mixed may be acquired due to factors such as the position of the cells in the container or the tilt between the container and the optical axis during imaging. Such images can also be used without any problems as training images and test images.

[0074] A third advantage of the image processing according to this embodiment is that it does not increase the burden on the user when enriching the case studies. As mentioned above, in addition to not needing to determine the contrast state of the collected images, image inversion can be performed automatically, and the instruction input for the original image can be applied directly to the inverted image. Therefore, the work that the user must do is essentially not increased at all.

[0075] The present invention is not limited to the above-described embodiment, and various modifications other than those described above are possible without departing from the spirit of the present invention. For example, the imaging device 1 of the above embodiment scans the imaging unit 13 across the sample container 10 to capture an image of the sample S. However, the imaging method is not limited to this. For example, an image captured using an optical microscope may be used as at least one of the original image and the test image. Furthermore, in the above embodiment, illumination light is incident from above the sample S and imaging is performed using transmitted light downward. However, the illumination and imaging directions are not limited to this and may be any as long as a bright-field image can be obtained.

[0076] In the above embodiment, for example, instruction input for instructing cellular regions in the acquired original image is accepted and labeling is applied to the original image. However, for example, if there is already an image for which instruction work has been performed in the past, that image and the instruction results may be used. In this case, it is possible to omit instruction input at least for such images.

[0077] Furthermore, in the above embodiment, the image processing of the present invention is performed by an imaging device 1 configured for imaging cells. However, the image processing of the present invention can also be performed by a device that does not itself have an imaging function, such as a computer device such as a personal computer or workstation. In this case, images can be acquired externally, for example, via the interface unit 142 and a telecommunications line, or via an appropriate storage medium. The present invention can also be embodied as a control program for causing a computer device to execute each step of the image processing method or classification model construction method of the present invention, or as a storage medium on which the program is non-temporarily recorded.

[0078] Furthermore, in the above embodiment, bright-field images of cells cultured on a flat surface are processed, but the image processing of this embodiment can also be applied to samples in which, as described above, two types of contrast states appear due to the cells acting as thin lenses.

[0079] As described above with reference to specific embodiments, in the image processing method and classification model construction method according to the present invention, the inverted image can be, for example, an image obtained by inverting the luminance of each pixel of the original image and then scaling the luminance value of each pixel so that the average luminance is equal to that of the original image. Alternatively, the inverted image can be, for example, an image obtained by inverting the luminance of each pixel of the original image and then scaling the luminance value of each pixel so that the average luminance in regions other than the cellular region is equal to that of the original image.

[0080] Furthermore, for example, the inverted image may be an image in which the brightness of each pixel in the designated cell region of the original image is inverted. In this case, the brightness value of each pixel in the cell region of the inverted image may be scaled so that the average brightness of the cell region after inversion is the same as the average brightness of the cell region in the original image. These methods make it possible to create an inverted image that serves as a teacher image that contributes to improving the generalization performance and classification accuracy of the classification model.

[0081] Here, the classification model may be constructed using deep learning. Images of cells have indefinite shapes and vary greatly from one to another. Deep learning algorithms, which do not require artificially determining the features to focus on for classification, are suitable for analyzing images with such characteristics.

[0082] For example, a step of inputting a bright-field image of two-dimensionally cultured cells as a test image into the constructed classification model and acquiring the output image may be further provided. In this configuration, regardless of whether the test image is captured under either of the two contrast conditions, it is possible to accurately extract a cell region from the test image. Therefore, the user can select the contrast condition during imaging according to their preference or purpose, without being concerned about the classification accuracy of the classification model.

[0083] Furthermore, the image processing method according to the present invention may further include a step of segmenting the test image into cell regions and other regions based on the output image of the classification model. With this configuration, once an image of cells is prepared, it becomes possible to automatically segment the image into cell regions and other regions.

[0084] For example, the original image can be an image of a sample in which cells of the same species as those for which the test image is captured are two-dimensionally cultured. With this configuration, an image captured under conditions similar to the image of the cells to be processed is used as the teacher image, making it possible to build a highly accurate classification model. [Industrial Applicability]

[0085] The present invention is particularly suitable for use in the medical and biological science fields, for example, for the purpose of observing and evaluating optical images of cultured cells, but its application fields are not limited to the medical and biological science fields. [Explanation of symbols]

[0086] 1. Imaging device 11 Holder 12 Lighting Department 13 Imaging unit 14 Control Unit 141 CPU 147 Input reception section 148 Display section C cells S sample

Claims

1. A step of acquiring a correct image indicating a cell region occupied by the cell in a plurality of original images obtained by bright-field imaging of the cell in the original images; creating an inverted image by inverting the brightness of at least the cell region based on each of the original images; a step of constructing a classification model by performing machine learning based on training data including a set of the original image and the corresponding correct answer image, and a set of the inverted image and the correct answer image corresponding to the original image on which the inverted image is based; An image processing method comprising:

2. 2. The image processing method according to claim 1, wherein the inverted image is an image obtained by inverting the luminance of each pixel of the original image and scaling the luminance value of each pixel so that the average luminance is equal to that of the original image.

3. 2. The image processing method according to claim 1, wherein the inverted image is an image obtained by inverting the brightness of each pixel of the original image and further scaling the brightness value of each pixel so that the average brightness in the region other than the cellular region is equal to that of the original image.

4. 2. The image processing method according to claim 1, wherein the inverted image is an image obtained by inverting the brightness of each pixel in the designated cell region of the original image.

5. 5. The image processing method according to claim 4, wherein the luminance value of each pixel in the cell region in the inverted image is scaled so that the average luminance of the cell region after inversion is the same as the average luminance of the cell region in the original image.

6. The image processing method according to claim 1 , wherein the classification model is constructed by deep learning.

7. 7. The image processing method according to claim 1, further comprising the step of inputting a bright-field image of two-dimensionally cultured cells as a test image into the constructed classification model and acquiring an output image thereof.

8. The image processing method according to claim 7 , further comprising the step of segmenting the test image into the cell region and a non-cell region based on the output image of the classification model.

9. 9. The image processing method according to claim 7, wherein the original image is an image of a sample in which cells of the same kind as the cells from which the test image is taken are two-dimensionally cultured.

10. A method for constructing a classification model for identifying an area occupied by a cell from a bright-field image of two-dimensionally cultured cells, comprising: A step of acquiring a correct image indicating a cell region occupied by the cell in a plurality of original images obtained by bright-field imaging of the cell in the original images; creating an inverted image by inverting the brightness of at least the cell region based on each of the original images; a step of constructing a classification model by performing machine learning based on training data including a set of the original image and the corresponding correct answer image, and a set of the inverted image and the correct answer image corresponding to the original image on which the inverted image is based; A method for building a classification model comprising:

Images