Image processing method and construction method of classification model

Abstract

The image processing method of the present application includes the following steps: for each of a plurality of original images obtained by bright field shooting of cells, a process of acquiring a correct answer image showing a cell region occupied by a cell within the original image; a process of creating an inverted image in which the brightness is inverted at least for the cell region based on each original image; a process of executing machine learning and constructing a classification model using as training data a set of an original image and a correct answer image corresponding to the original image, and a set of an inverted image and a correct answer image corresponding to an original image that is the basis of the inverted image; and a process of inputting a bright field image of a two-dimensionally cultured cell as a test image into the classification model and acquiring an output image thereof. According to the present application, it is possible to construct a classification model that can accurately extract a cell region in an image from either of two bright field images having different contrast states.

Description

Technical Field

[0001] This invention relates to an image processing method for determining the region occupied by cells from an image obtained by photographing cells, and a method for constructing a classification model for performing the method using a machine learning algorithm. Background Technology

[0002] In medical or biological science experiments, cells are photographed using CCD cameras and other similar devices, and the images are digitized. Various image processing techniques are then applied to this image data for observation or analysis. For example, multiple images of cells are collected as training images and fed together with corrective images that demonstrate the regions occupied by cells in these images to a classification model for machine learning. This allows the construction of a classification model capable of automatically extracting the regions occupied by cells from newly provided test images.

[0003] Images obtained from photographing cells are bright-field images, such as those taken using an optical microscope. Because cells are nearly transparent, the image of the cells may not be clear in a focused image. Conversely, images taken under slightly off-focus conditions sometimes show good cell visibility. For example, when photographing cells in two-dimensional culture (planar culture), it is known that the contrast between the cell's edge and inner portion reverses depending on the focal point position during photography. Specifically, there are cases where the cell's edge is bright (high brightness) and the inner portion is dark (low brightness) (hereinafter referred to as the "first contrast state") and conversely, where the edge is dark and the inner portion is bright (hereinafter referred to as the "second contrast state") (see, for example, Japanese Patent Application Laid-Open No. 2018-163248).

[0004] Thus, it can be said that both types of images with different contrasts are suitable for cell observation. Which type of image to acquire depends primarily on the photographer's preference and the environment at the time of shooting. Furthermore, depending on the state of the sample at the time of shooting, the two contrast states mentioned above may also coexist in the image.

[0005] Multiple such acquired images can be collected and used as training images for constructing a classification model that automatically extracts the aforementioned cell regions. In this case, the contrast state of each training image can be either the first contrast state or the second contrast state described above. Furthermore, the contrast state of the test image, which becomes the object of the automatic extraction process, can also be either the first contrast state or the second contrast state.

[0006] If the training images used in machine learning include many images taken at the same contrast levels as the test images, then automatic extraction based on the classification model can be expected to perform with high accuracy. However, if this is not the case, the extraction accuracy is further reduced. Furthermore, it is preferable to achieve equal extraction accuracy regardless of the contrast level of the test images.

[0007] Therefore, as training images, images captured under both contrast states are preferably collected unbiasedly. When new shots are taken while preparing training images, this can be addressed by taking this into account during the shooting process. However, the workload of capturing multiple samples simultaneously to collect training images while achieving both focus states becomes very large. On the other hand, if previously captured images can be used as a library, the workload can be significantly reduced. However, it is not always possible to prepare a library of images that unbiasedly includes both contrast states. Summary of the Invention

[0008] The present invention was made in view of the above-mentioned problems, and its object is to provide a technique for constructing a classification model that can extract cellular regions from either of two bright field images with different contrast states with high precision.

[0009] To achieve the above objectives, one aspect of the present invention is an image processing method comprising: a step of acquiring, for each of a plurality of original images obtained by taking a bright-field photograph of a cell, a forward image showing the cell region occupied by the cell within the original image; a step of creating, based on each of the original images, a reverse image that inverts the brightness at least for the cell region; and a step of performing machine learning and constructing a classification model by using the original image and a set of forward images corresponding to the original image, and the reverse image and a set of forward images corresponding to the original image as the basis of the reverse image, respectively, as training data.

[0010] Another aspect of the present invention relates to a method for constructing a classification model for determining the region occupied by a cell from a bright-field image of a two-dimensional cultured cell. To achieve the above objective, the method for constructing the classification model includes: for each of a plurality of original images obtained by bright-field imaging of the cell, acquiring a forward image showing the cell region occupied by the cell within the original image; based on each of the original images, creating a reverse image that inverts the brightness at least for the cell region; and using the original image and a set of forward images corresponding to the original image, as well as the reverse image and a set of forward images corresponding to the original image as the basis of the reverse image, respectively, as training data, performing machine learning, and constructing a classification model.

[0011] In this invention, regardless of which of the two contrast states the original images collected for training purposes are in, a classification model capable of extracting cellular regions from images with high accuracy can be constructed. Furthermore, good extraction results can be obtained regardless of which of the two contrast states the input image, as the object of cellular region extraction, is input into the classification model. The reasons are as follows.

[0012] As described above, there are two contrast states in the images obtained by taking bright-field photographs of cells. Specifically, in the first contrast state, the edge (outline) portion of the cell is bright, while the inner portion becomes darker. Conversely, in the second contrast state, the edge portion becomes darker, while the inner portion becomes brighter. Thus, the image contrast relationship between the edge and inner portions of the cell is symmetrical between the two contrast states.

[0013] Therefore, if the brightness of an image is inverted, it is expected that the two contrast states can be transformed into each other; that is, an inverted image representing the other contrast state can be created from the original image of one contrast state. Such a simple inverted image may not be suitable for cell observation. However, experiments conducted by the inventors of this application have confirmed that it is very effective as a training image for constructing a classification model.

[0014] The original image is obtained to construct a classification model that automatically extracts cell regions from bright-field images. It is a paired image with the forward image that teaches the cell regions in the original image. Furthermore, in principle, the areas occupied by cells are the same in both the original and reverse images. That is, the forward image also teaches the cell regions in the reverse image.

[0015] Therefore, sets of original and forward-resolved images, as well as sets of inverted and forward-resolved images, can be treated as training data corresponding to the same cells but with different contrast states. This enriches the examples used in the training data and allows for the unbiased collection of examples under both contrast states. By performing machine learning using the training data collected in this way, the constructed classification model exhibits generalization performance capable of handling input images with any contrast state. In other words, it is possible to construct a classification model that can extract cell regions from any input image with high accuracy.

[0016] Furthermore, this contrast reversal occurs within the cellular region. Therefore, when creating an inverted image, it is sufficient to reverse the brightness only in at least the cellular regions of the original image. Since the cellular regions in the original image are taught to the correct image, it is also possible to selectively invert only the cellular regions of the original image.

[0017] Furthermore, the method described above does not distinguish between the contrast levels used when the original and input images were captured. Therefore, it is not necessary to limit the contrast levels used when collecting training images and capturing input images. Additionally, even images with a mixture of two contrast levels can be used without any problems as the original images for building the classification model and as the input images for the constructed classification model.

[0018] According to the present invention, regions at least taught to be cellular regions in the original image acquired as training images are inverted, and the inverted image and the original image are used together as training images to perform machine learning. Thus, a classification model capable of extracting cellular regions from an image with high accuracy from either of two bright field images with different contrast states can be constructed. Furthermore, the classification model constructed in this way can extract cellular regions from an input image with high accuracy regardless of the contrast state of the input image. Attached Figure Description

[0019] Figure 1 This is a diagram showing a schematic structure of an embodiment of an imaging apparatus capable of performing the image processing method involved in the present invention.

[0020] Figure 2 This is a flowchart illustrating the image processing in this embodiment.

[0021] Figures 3A to 3C This is an example of a bright-field image representing cells cultured in a plane.

[0022] Figure 4 This is a flowchart illustrating the method for constructing the classification model in this embodiment.

[0023] Figure 5It is a diagram that schematically represents the relationship between the original image, the inverted image, and the forward image.

[0024] Figure 6 This is a flowchart illustrating the first example of image reversal processing.

[0025] Figure 7 This is a flowchart illustrating the second example of image reversal processing.

[0026] Figure 8 This is a flowchart illustrating the fourth example of image reversal processing.

[0027] Figure 9A and Figure 9B This is a diagram showing examples of original and reversed images.

[0028] Figure 10A and Figure 10B This is a diagram showing an example of the extraction results of the cell region involved in this embodiment.

[0029] Explanation of reference numerals in the attached figures

[0030] 1. Filming device

[0031] 11 Brackets

[0032] 12 Lighting Department

[0033] 13 Filming Department

[0034] 14 Control Department

[0035] 141 CPU

[0036] 147 Input Receiving Unit

[0037] 148 Display Department

[0038] C cells

[0039] S sample Detailed Implementation

[0040] Figure 1 This diagram illustrates a schematic structure of an imaging apparatus capable of performing the image processing method of the present invention. The imaging apparatus 1 is used to photograph a sample S, such as cells, contained in a sample container 10. The sample container 10 is flat and has an open top, and is sometimes referred to as a dish. A predetermined amount of liquid, serving as a culture medium M, is injected into the sample container 10. The sample S, such as cells, cultured in this liquid under predetermined culture conditions, serves as the object of imaging by the imaging apparatus 1. The culture medium can be a culture medium with appropriate reagents added, or a culture medium that gels after being injected into the sample container 10 in liquid form.

[0041] In addition to the above, tissue sections or pathological specimens supported on a suitable carrier can also be used as subjects for imaging. Alternatively, the sample container can be a so-called perforated plate with multiple recesses called traps, with the sample held in each trap serving as the subject for imaging.

[0042] The imaging device 1 includes a support 11 for holding the sample container 10, an illumination unit 12 disposed above the support 11, an imaging unit 13 disposed below the support 11, and a control unit 14 having a CPU 141 for controlling the operation of these parts. The support 11 abuts against the periphery of the lower surface of the sample container 10, holding the sample container 10 in a generally horizontal position.

[0043] The illumination unit 12 illuminates the sample container 10 held by the support 11 with illumination light. A white LED (Light Emitting Diode) can be used as the light source. The illumination unit 12 is a combination of a light source and a suitable illumination optics system. The object being photographed inside the sample container 10 is illuminated from above by the illumination unit 12.

[0044] The imaging unit 13 is positioned below the sample container 10 held by the bracket 11. An imaging optical system is arranged directly below the sample container 10 in the imaging unit 13. The optical axis of the imaging optical system faces vertically. Figure 1 This is a side view; the vertical direction is indicated by the top and bottom.

[0045] The imaging unit 13 images the object to be photographed inside the sample container 10. Specifically, light emitted from the illumination unit 12 and incident on the liquid above the sample container 10 illuminates the object. Light passing downwards from the bottom of the sample container 10 is incident on the light-receiving surface of the imaging element 132 via the imaging optical system of the imaging unit 13, which includes the objective lens 131. The image of the object to be photographed, formed on the light-receiving surface of the imaging element 132 by the imaging optical system, is captured by the imaging element 132. The imaging element 132 is a region image sensor with a two-dimensional light-receiving surface, such as a CCD sensor or a CMOS sensor.

[0046] The imaging unit 13 can move horizontally and vertically via a mechanical control unit 146 provided in the control unit 14. Specifically, the mechanical control unit 146 activates the drive mechanism 15 based on control commands from the CPU 141, causing the imaging unit 13 to move horizontally. Therefore, the imaging unit 13 moves horizontally relative to the sample container 10. Furthermore, focus adjustment is performed by moving vertically.

[0047] As shown by the dashed arrow in the figure, when the driving mechanism 15 moves the imaging unit 13 in the horizontal direction, the illumination unit 12 moves integrally with the imaging unit 13. That is, the illumination unit 12 is configured so that its optical center is approximately aligned with the optical axis of the imaging unit 13, and the illumination unit 12 moves in conjunction with the imaging unit 13 when the imaging unit 13 moves in the horizontal direction. Therefore, even when the imaging unit 13 moves relative to the sample container 10, the optical center of the illumination unit 12 is always located on the optical axis of the imaging unit 13. As a result, the illumination conditions for the object being photographed can be kept constant regardless of the position, thereby maintaining good imaging conditions.

[0048] The image signal output from the imaging element 132 of the imaging unit 13 is sent to the control unit 14. That is, the image signal is input to the AD converter (A / D) 143 provided in the control unit 14 and converted into digital image data. The CPU 141 performs appropriate image processing based on the received image data.

[0049] The control unit 14 also includes a memory 144 for temporarily storing image data and data in operation, and a storage device 145 for storing programs to be executed by the CPU 141 and data created by the CPU 141. The memory 144 can be accessed by the CPU 141 at high speed, but its storage capacity is smaller than that of the storage device 145. The storage device 145 is, for example, a hard disk drive (HDD), which has a large storage capacity but a slower access speed than the memory 144. They are used separately depending on their purpose. The CPU 141 performs various computational processes, described later, by calling and executing the control program stored in the storage device 145.

[0050] Furthermore, an interface (IF) unit 142 is provided in the control unit 14. In addition to user interface functions such as receiving operation input from the user and displaying processing results to the user, the interface unit 142 also has the function of exchanging data with external devices connected via a communication line. To implement the user interface functions, an input receiving unit 147 for receiving operation input from the user and a display unit 148 for displaying messages output to the user or processing results are connected to the interface unit 142.

[0051] Next, an image processing method using the imaging device 1 configured as described above will be explained. This image processing method corresponds to one embodiment of the image processing method of the present invention. The purpose of this processing is to automatically extract the area occupied by the cells from a bright-field image captured by the imaging device 1 along the bottom surface of the container 10, using cells C that have undergone planar culture (two-dimensional culture) as a sample S.

[0052] Figure 2This is a flowchart illustrating the image processing in this embodiment. This processing is achieved by the CPU 141 of the imaging device 1 executing a pre-prepared control program and causing each part of the device to perform predetermined actions. First, a bright-field image of the sample S is acquired (step S101). Specifically, a sample container 10, on which cells serving as the sample S are cultured in two dimensions, is mounted on the support 11 of the imaging device 1. The imaging unit 13 of the imaging device 1 scans the sample container 10 while simultaneously capturing an image of the sample container 10, thereby acquiring a bright-field image of the sample S.

[0053] The acquired bright-field image is input into a classification model pre-built using machine learning (step S102). As a result of machine learning, this learning model has the ability to identify regions occupied by cells (hereinafter referred to as "cell regions") and regions outside of them (hereinafter referred to as "background regions") from the bright-field image input as a test image, as detailed later. For example, a mask image used to segment the test image region into cell regions and background regions is output as the result image.

[0054] Based on the output image of the classification model, the original bright field image region is segmented into a cell region and a background region (steps S103 and S104). That is, by applying the mask image output by the classification model to the original bright field image, only the region occupied by cells in the bright field image is extracted. Thus, the cell region and the background region are separated.

[0055] Next, the method for constructing the classification model used to implement the above image processing will be described. Such classification models are widely used for the purpose of extracting regions with specific appearance features from test images. Various machine learning algorithms are known as suitable for constructing classification models for this purpose, and appropriate ones can be selected and utilized in this embodiment. In the analysis of cell images with irregular shapes and large individual differences, deep learning algorithms that do not require manually setting the feature quantities for classification can be appropriately applied. Here, a convolutional neural network (CNN) is used as an example, but it is not a limitation.

[0056] The construction of a machine learning-based classification model involves collecting typical examples as training images, labeling these training images (teaching inputs), and performing machine learning on them as training data. Therefore, it is necessary to collect images of cells with the same or similar characteristics as the images of cells being classified. For example, if a library of previously captured images exists, that library can be utilized. However, when dealing with bright-field images of planar cultured cells, the following aspects need to be considered.

[0057] Figures 3A to 3CThis is an example diagram showing a bright-field image of cells cultured in a plane. Figure 3A and Figure 3B During the imaging process, the focal position relative to the sample varied slightly. Therefore, the contrast of the cell images differed. Specifically, in... Figure 3A In image Ia, the cell outline (edge) appears white, while the cell interior appears darker. That is, in image Ia, the cell outline has higher brightness, while the cell interior has relatively lower brightness. On the other hand, in images like... Figure 3B In image Ib, contrary to the above, the cell outline appears darker, while the cell interior appears whiter. That is, in image Ib, the cell outline has lower brightness, while the cell interior has relatively higher brightness. This difference is thought to be due to the cells in planar culture spreading thinly along the container wall and acting like a convex lens.

[0058] like Figure 3C As shown, image Ia was obtained when the focal plane Fa of objective lens 131 was slightly inward of cell C. Similarly, image Ib was obtained when the focal plane Fb of objective lens 131 was slightly forward of cell C. These conditions, strictly speaking, deviate from the focusing conditions for cell C, but are still suitable for observing cell C. This is because, since cell C is nearly transparent, its visibility is actually lower in a fully focused image. However, in images Ia and Ib, where the focal position is slightly off, the image of cell C appears clearly.

[0059] Thus, two images with different contrast states can be used as bright-field images suitable for observing cells in two-dimensional culture. Specifically, in the image of the first contrast state, such as Figure 3A As shown in image Ia, the outline of the cell is bright, while the central part is dim. On the other hand, in the image of the second contrast state, as... Figure 3B As shown in image Ib, the cell outlines are low-brightness, while the central areas are high-brightness. There is essentially no inherent superiority of one over the other; the choice depends primarily on the observer's preference, purpose, and the environment in which the image was taken.

[0060] Among the examples collected as candidates for training images, there may be images with both contrast states. There may be images biased towards either contrast state, or a mixture of both. Furthermore, the test images used for image processing may also have both contrast states.

[0061] Training images are preferably images captured with the same contrast as the test images. However, especially when using a database of previously captured images, it may not be possible to collect only the required number of such examples. Moreover, distinguishing multiple images based on this contrast is a significant burden for the operator. Furthermore, when the test images are captured with a different contrast than the training images, sufficient classification accuracy cannot be obtained. Moreover, depending on the location of cells within the sample S, the shooting environment, etc., two contrast states may coexist in a single image.

[0062] Therefore, the training images used as training data preferably include images taken in both contrast states to an equal extent. However, it remains constant that the required number of such images cannot always be prepared.

[0063] Therefore, the inventors of this application, focusing on the characteristic that the image contrast of cells is symmetrical in the two types of images mentioned above, arrived at the following concept: That is, by inverting the contrast of images collected as training images, it is possible to virtually create an image of another contrast state from an image of one contrast state. Furthermore, by using both of these two contrast state images as training images, the training examples can be enriched.

[0064] By using images collected in this way under two contrast states as training images, it is expected that a classification model with equivalent classification accuracy, i.e., excellent generalization performance, can be constructed for test images under any contrast state. Furthermore, according to the inventors' verification experiments, it has been confirmed that a classification model with sufficient classification accuracy for test images under any contrast state can be obtained by using training images virtually created in this manner. The specific method for constructing such a classification model will be described below.

[0065] Figure 4 This is a flowchart illustrating the method for constructing the classification model in this embodiment. Additionally, Figure 5 This is a diagram schematically illustrating the relationship between the original image, the inverted image, and the forward image. This processing can be implemented by executing a pre-prepared control program via the CPU 141 of the imaging device 1. However, without the need for new captures to acquire training images, this processing can be performed using a general-purpose computer device with a typical hardware architecture. Since high computing power is required to execute machine learning algorithms, it is preferable to use a computer device with a higher performance than the control unit 14 provided in the imaging device 1.

[0066] First, an original image Io that can be used as a training image is acquired (step S201). The original image Io can be a bright-field image of cells of the same species as the sample S, which is the object of image processing, but is independent of the contrast state at the time of capture. That is, it can be any one of the first contrast state, the second contrast state, any one of them, or a mixture thereof. The ratio is also arbitrary. More preferably, the cells in the original image Io are two-dimensional cultured cells identical to those in the sample S. The original image Io can be newly captured by the capturing device 1 or acquired from an image library that collects previously captured images.

[0067] Next, teaching input from the user is received for the original image Io (step S202). This teaching input is an operation for marking the areas occupied by cells in the image. For example, the original image Io can be displayed on the display unit 148, and teaching input can be received through user input via the input receiving unit 147.

[0068] Based on the teaching results, a forward image It is created (step S203). The forward image It is an image in which the original image Io is clearly divided into the cellular region occupied by the cells and the background region outside of them. For example, an image obtained by binarizing the brightness in the cellular region and the background region can be used as the forward image It. When only the cellular region (or background region) is extracted from the original image Io, this binarized image can be used as a mask image.

[0069] Next, an inverted image Ir is created based on the original image Io (step S204). The inverted image Ir is an image whose contrast is changed by inverting the brightness of each pixel in at least the cell region of the inverted original image Io. The specific processing method will be described later. When the original image Io is an image of cells obtained by taking pictures in a first contrast state, the inverted image Ir virtually represents an image obtained by taking pictures of the same cells in a second contrast state. On the other hand, when the original image Io is an image of cells obtained by taking pictures in a second contrast state, the inverted image Ir virtually represents an image obtained by taking pictures of the same cells in a first contrast state. Through this processing, a set of original images Io and their corresponding forward images It and inverted images Ir is created. By performing the same processing on multiple collected original images Io, multiple sets of original images Io, their corresponding forward images It, and inverted images Ir are created.

[0070] Both the original image Io and the reversed image Ir are used as training images, thereby enriching the training images. Specifically, the original image Io and its corresponding forward image It are paired to form a set of training data D1 (step S205). Additionally, the reversed image Ir and its corresponding forward image It are paired to form another set of training data D2 (step S206). The original image Io and the reversed image Ir are images obtained by capturing the same cells in the same field of view with different contrast conditions. Therefore, the cell regions taught in the original image Io are also automatically taught to the cell regions in the reversed image Ir. That is, the forward image It for the original image Io is also located as the forward image for the reversed image Ir.

[0071] In this way, two training images (original image Io and inverted image Ir) are prepared from a single original image Io, and the teaching task only needs to be performed on one image. Therefore, the user's workload does not increase while enriching the training data.

[0072] Based on the prepared training data, machine learning is performed using an appropriate learning algorithm (step S207). As described above, a learning algorithm based on deep learning principles can be appropriately applied. Furthermore, as a method for region segmentation of the image, well-known semantic segmentation methods can be applied, for example.

[0073] The classification model constructed through the above processing can segment any test image into cell regions and background regions for both the first and second contrast states. When the binarized mask image is used as the forward image, the output image of the classification model also becomes the mask image used to segment the test image into cell regions and background regions. By applying this mask image to the test image and performing appropriate image processing, it is possible to distinguish between cell regions and background regions within the test image. Examples of image processing in this case include, for instance, cutting out either cell regions or background regions from the image, assigning different visual effects (e.g., color distinction) to cell regions and background regions, and emphasizing the contours of cell regions; however, the processing is not limited to these.

[0074] Next, refer to Figures 6 to 8 The processing used to create an inverted image from the original image ( Figure 4 Several examples of step S204 will be explained here. Here, four specific examples will be explained, but in the purpose of creating virtual training images based on the original images, the same result can be obtained through any one of these examples.

[0075] Figure 6This is a flowchart illustrating the first example of the image inversion process. In this example, the overall brightness of the original image is inverted (step S301). Therefore, the image contrast of the cell regions is inverted. However, since the background region is also inverted, there is a situation where the overall image density differs significantly from the actual density. Therefore, the average brightness of the original image and the inverted image are calculated (steps S302, S303), and to make them approximately consistent, the brightness values ​​of each pixel constituting the inverted image are scaled (step S304). Thus, an inverted image is created.

[0076] In this process, cell regions and background regions in the original image are not distinguished. Therefore, after acquiring the original image, a reversed image can be produced immediately without waiting for teaching input. In contrast, in the examples described below, cell regions and background regions in the original image are distinguished in advance. In this embodiment, since such distinction can be made through teaching input from the user, processing based on this premise is possible.

[0077] Figure 7 This is a flowchart illustrating a second example of the image inversion process. In this example, similar to the first example, the overall brightness of the original image is inverted (step S401). However, brightness scaling is performed only on the background region. That is, the average brightness of the background regions of both the original and inverted images is calculated (steps S402, S403), and to make their average brightness approximately equal, the brightness values ​​of each pixel constituting the background region of the inverted image are scaled (step S404). The inverted image is thus created.

[0078] By inverting the brightness, image contrast is reversed only in the cellular regions, but the overall brightness changes drastically in the roughly identical background regions. However, in actual shooting, background density does not change drastically due to minor differences in focus setting. Therefore, in this example, only the brightness of the inverted background regions is scaled. Thus, image contrast in the cellular regions is maintained. Furthermore, since the brightness of the background regions in the inverted image is not affected by the brightness of the cellular regions, the difference in brightness between the background regions before and after inversion can be reduced.

[0079] Next, a third example of image inversion processing will be described. In this example, in the original image, only the brightness of the regions taught as cell areas is inverted. Therefore, even in the inverted image, the background area remains unchanged, and the inverted image can be directly used as the inverted image without scaling. Since only the brightness of the cell regions is inverted, the flowchart illustration is omitted.

[0080] Figure 8This is a flowchart illustrating the fourth example of image inversion processing. In this example, similar to the third example, only the brightness of the cell regions is inverted (step S501), while the brightness of the background regions remains unchanged. However, the image contrast is adjusted by scaling the brightness of the inverted cell regions. That is, the average brightness of the cell regions in the original image and the inverted image are calculated separately (steps S502 and S503), and the brightness values ​​of each pixel constituting the inverted cell regions are scaled in a manner that makes them approximately equal (step S504).

[0081] Depending on the cell condition or lighting conditions, the brightness or image contrast of the cell area can sometimes appear unnatural when simply inverting the image. This problem can be eliminated by scaling the inverted cell area.

[0082] As step S204 ( Figure 4 The process of creating an inverted image can use any of these examples. Alternatively, it can be configured to switch between them as needed. For example, the processing method can be selected through user operation. Furthermore, the scaling in the examples above is based on the average brightness, but this is only one example. For example, scaling based on the median brightness, scaling based on the brightness difference (or contrast value) between the cell region and the background region, etc., can also be performed.

[0083] Figure 9A and Figure 9B This is a diagram showing examples of original and reversed images. Figure 9A The image shown, Ic, is the original image; the outlines of the cells appear brighter, while the interiors of the cells appear darker. This image was taken at the first contrast setting. On the other hand, Figure 9B The image ID shown is obtained through the first example mentioned above ( Figure 6 This example illustrates a method for creating an inverted image from image Ic. It can be seen that the cell outlines are darker, while the cell interiors are brighter, indicating a reversal of image contrast. On the other hand, the background area shows almost no change in density compared to the original image Ic. This image Id virtually represents the image that would be obtained if the image were taken under the second contrast condition.

[0084] Figure 10A and Figure 10B This is a diagram showing an example of the extraction results of the cell region involved in this embodiment. Figure 10AThis is a comparative example. Image Ie represents the result of extracting cell regions from a test image using an output image obtained by inputting a test image taken at a second contrast level into a classification model built solely using images taken at a first contrast level as training images. Although multiple cells are distributed in image Ie, the portion enclosed by the dashed line is not extracted as a cell region. Discrepancies exist between the actual image and the extraction result, such as entire cells being missed from the extraction target, or partial regions of cells not being extracted, indicating insufficient extraction accuracy.

[0085] on the other hand, Figure 10B The image If shown represents the extraction result of a classification model constructed using the original image and its inverted representation as training images. It can be seen that the area occupied by the cells in the image closely matches the extracted area, further improving extraction accuracy.

[0086] As described above, in the image processing of this embodiment, the classification model used to analyze the bright-field image of cells in planar culture (two-dimensional culture) and extract the cell regions occupied by the cells is constructed by machine learning using both the collected original image and the inverted image made from it as training images. This structure has the following advantages.

[0087] First, by enriching the examples used for training, the generalization performance and classification accuracy of the classification model can be improved. Specifically, regardless of whether the image of the cell being processed was captured in the first or second contrast state, the cell region can be extracted from the image with high accuracy.

[0088] Second, when performing this image processing, it is not necessary to distinguish the contrast level at which the image was captured. This also includes the following implications.

[0089] First, regardless of the contrast level at which the images used as training images were captured, they contribute to the enrichment of examples. Therefore, the ratio of contrast levels among the collected images can be biased, or in extreme cases, they can be images with only any contrast level. The contrast level at the time of capture is often chosen based on the observer's preference. Thus, images collected, for example, within a research facility, may sometimes be biased towards any particular contrast level. Even in such cases, enriching the examples can be achieved by virtually creating images equivalent to another contrast level and using them along with the original images as training images. This is particularly advantageous when using a library of previously captured images as training images.

[0090] Furthermore, it is unnecessary to determine the contrast level at which each image was captured. Even if images from two contrast levels coexist, no differentiation processing is required when using them as training images; no technical effort is needed for this. Therefore, the collected images can be used as training images without waste or unnecessary effort.

[0091] Furthermore, the contrast level at which the image was captured is irrelevant to the image being processed. Therefore, users don't need to worry about the convenience of image processing and can take photos at a contrast level appropriate to their purpose or preference. Additionally, even when processing images that have already been taken, good processing results can be obtained regardless of their contrast level, and the user doesn't need to be aware of this.

[0092] Furthermore, even images where both contrast states exist in a single image can be used as the original image or the image to be processed in this embodiment. Due to factors such as the position of the cells within the container and the tilt of the container relative to the optical axis during imaging, images with both contrast states present can be obtained. Such images can also be used without any problems as training or testing images.

[0093] As a third advantage of the image processing involved in this embodiment, it can be cited that the burden on the user is not increased when the number of implementation examples is enriched. As described above, except that it is not necessary to determine the contrast state of the collected image, image inversion can be performed as an automatic process. In addition, the teaching input of the original image can be directly applied to the inverted image. Therefore, the work that the user needs to do is not increased at all.

[0094] Furthermore, the present invention is not limited to the embodiments described above, and various modifications other than those described above can be made without departing from its spirit. For example, the imaging device 1 in the above embodiment is an apparatus in which the imaging unit 13 scans the sample container 10 to image the sample S. However, the imaging method is not limited to this. For example, an image taken using an optical microscope can be used as at least one of the original image and the test image. In addition, in the above embodiment, illumination light is incident from above the sample S, and imaging is performed using transmitted light transmitted downwards. However, the direction of illumination and imaging is not limited to this, and can be arbitrary as long as a bright field image can be obtained.

[0095] Additionally, for example, in the above embodiment, teaching input is received for the acquired original image to teach cell regions, thereby labeling the original image. However, for example, if an image has already been used for teaching in the past, that image and the teaching result can also be utilized. In this case, at least for such images, the teaching input can be omitted.

[0096] Furthermore, the above-described embodiment is an embodiment in which the image processing of the present invention is performed by an imaging device 1 having a structure for imaging cells. However, the image processing of the present invention can also be performed by a device that does not have its own imaging function, such as a computer device like a personal computer or workstation. In this case, for example, images can be acquired from the outside via an interface 142 and a telecommunication line or via a suitable storage medium. Moreover, the present invention can be implemented as a control program for causing a computer device to execute each step of the image processing method or classification model construction method of the present invention, or as a storage medium that non-temporarily stores the program.

[0097] Furthermore, in the above embodiment, the bright-field image of planar cultured cells is used as the processing object. However, similarly to the above, the image processing of this embodiment can be applied to samples exhibiting two contrast states due to the cells acting as thin lenses.

[0098] As illustrated above with specific embodiments, in the image processing method and classification model construction method of the present invention, regarding the inverted image, for example, it can be created by inverting the brightness of each pixel in the original image and scaling the brightness value of each pixel to make the average brightness equal to that of the original image. Alternatively, for example, the inverted image can also be created by inverting the brightness of each pixel in the original image and scaling the brightness value of each pixel to make the average brightness in areas other than the cell region equal to that of the original image.

[0099] For example, an inverted image can also be created by inverting the brightness of each pixel within the taught cell region of the original image. In this case, to ensure that the average brightness of the inverted cell region is the same as the average brightness of the cell region in the original image, the brightness values ​​of each pixel within the cell region of the inverted image can be scaled. Based on these methods, inverted images, such as training images, can be created to help improve the generalization performance and classification accuracy of the classification model.

[0100] Here, the classification model can also be built using deep learning. Cellular images have variable shapes and exhibit significant individual differences. Deep learning algorithms, which do not require pre-determining the features to focus on for classification, are suitable for parsing images with such characteristics.

[0101] Alternatively, for example, a step can be set up whereby a bright-field image of two-dimensional cultured cells is input as a test image into a constructed classification model to obtain the output image. In this invention, regardless of whether the test image was captured in one of two contrast states, the cell region can be extracted from the test image with high accuracy. Therefore, the user can select the contrast state at the time of capture based on preference or purpose without being aware of the classification accuracy in the classification model.

[0102] Furthermore, in the image processing method of the present invention, a step can be included: based on the output image of the classification model, segmenting the test image region into cell regions and regions outside the cell regions. According to this structure, when preparing an image of cells, the image can be automatically segmented into cell regions and regions outside the cell regions.

[0103] Furthermore, for example, the original image can be set as an image taken from a sample obtained by two-dimensional culture of cells of the same species as the cells in which the test image is taken. Based on this structure, a high-precision classification model can be constructed by using images taken under conditions similar to those of the cells being processed as training images.

[0104] Industrial availability

[0105] This invention is particularly suitable for applications such as the medical and bioscience fields, for observing and evaluating optical images of cultured cells. However, its applications are not limited to the medical and bioscience fields.

Claims

1. An image processing method, wherein, including: a process of acquiring, for each of a plurality of original images obtained by bright field imaging of cells, a correct answer image in which a cell region occupied by the cells is taught within the original image; a process of creating, based on each of the original images, an inverted image in which the brightness is inverted at least for the cell region; a process of executing machine learning and constructing a classification model using, as training data, a group of the original images and the correct answer images corresponding to the original images, and a group of the inverted images and the correct answer images corresponding to the original images that are the basis of the inverted images, respectively.

2. The image processing method according to claim 1, wherein the inverted image is created by inverting the brightness of each pixel of the original image, and further scaling the brightness value of each pixel in such a manner that the average brightness is equal to that of the original image.

3. The image processing method according to claim 1, wherein the inverted image is created by inverting the brightness of each pixel of the original image, and further scaling the brightness value of each pixel in such a manner that the average brightness in a region other than the cell region is equal to that of the original image.

4. The image processing method according to claim 1, wherein the inverted image is created by inverting the brightness of each pixel within the cell region taught in the original image.

5. The image processing method according to claim 4, wherein the brightness value of each pixel within the cell region in the inverted image is scaled in such a manner that the average brightness of the inverted cell region is the same as that of the cell region in the original image.

6. The image processing method according to any one of claims 1 to 5, wherein the classification model is constructed by deep learning.

7. The image processing method according to any one of claims 1 to 5, wherein, further including: a process of inputting a bright field image of two-dimensionally cultured cells as a test image into the constructed classification model, and acquiring an output image thereof.

8. The image processing method of claim 7, wherein, further including: a process of segmenting the test image region into the cell region and a region other than the cell region based on the output image.

9. The image processing method according to claim 7, wherein the original image is an image obtained by imaging a specimen of cells two-dimensionally cultured in the same manner as the cells of which the test image is imaged.

10. A method of constructing a classification model for determining an area occupied by a cell from a bright field image of a two-dimensional cultured cell, wherein, including: a process of acquiring, for each of a plurality of original images obtained by bright field imaging of cells, a correct answer image in which a cell region occupied by the cells is taught within the original image; a process of creating, based on each of the original images, an inverted image in which the brightness is inverted at least for the cell region; a process of executing machine learning and constructing a classification model using, as training data, a group of the original images and the correct answer images corresponding to the original images, and a group of the inverted images and the correct answer images corresponding to the original images that are the basis of the inverted images, respectively.

Images