Cell image processing system, and cell image processing method

Abstract

To solve the problem that it is difficult for a user to search major cells from among a great amount of cells manually in a case that cell data to be an analysis object often involve a great amount of cells being more than thousands of cells.SOLUTION: A cell image processing system includes an image acquisition part for acquiring a cell image with a sample including a plurality of cells as a subject, a cell region extraction part for extracting a plurality of cell regions corresponding to the cells from the cell image, a cell information acquisition part for acquiring morphological information of the cell regions and characteristic information configured to include brightness information of the cell regions, a representative cell determination part for determining a representative cell region among the plurality of extracted cell regions on the basis of at least one or more kinds of information in information constituting the characteristic information, and a display control part for displaying image information of a cell region corresponding to the representative cell, position information corresponding to the representative cell region, and the characteristic information of a cell region corresponding to the representative cell in a display part.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a cell image processing system and a cell image processing method.

Background Art

[0002] It is often performed to acquire analysis data of cells obtained by cell image analysis and have a user confirm and analyze the acquired analysis data. Specifically, in order to grasp the overall image of the distribution of the analysis data, by visualizing it in a format such as a scatter plot, the tendency of the cell data group is quantitatively grasped. Furthermore, by comparing and confirming the individual cell data within the visualized data distribution with the state of the cells in the observation image, while overlooking the overall tendency as quantitative data, operations such as confirming the state of living cells are also performed.

[0003] In cell image analysis software that supports such operations, the user picks up cells of interest while confirming the cells in the observation image. And it is possible to list up the crop image including the corresponding cells and the cell data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 discloses an image processing apparatus that extracts feature amounts from a slice image and superimposes a feature amount image including information on the extracted feature amounts on the slice image and displays it on a display unit.

[0006] However, in many cases, the cell data to be analyzed involves thousands or more cells. In such cases, it is difficult for the user to manually search for the main cells among the large number of cells. The user needs to select the slice image, which imposes a heavy burden on the user.

Means for Solving the Problems

[0007] The cell image processing system according to the present invention includes an image acquisition unit that acquires a cell image of a sample including a plurality of cells as a subject, a cell region extraction unit that extracts a plurality of cell regions corresponding to cells from the cell image, a cell information acquisition unit that acquires characteristic information including the morphological information of the cell regions and the luminance information of the cell regions, and a representative cell determination unit that determines a representative cell region among the plurality of extracted cell regions based on at least one or more types of information among the information constituting the characteristic information, and a display control unit that causes the display unit to display the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell.

[0008] The cell image processing method according to the present invention includes an image acquisition step of acquiring a cell image of a sample including a plurality of cells as a subject, a cell region extraction step of extracting a plurality of cell regions corresponding to cells from the cell image, a cell information acquisition step of acquiring characteristic information including the morphological information of the cell regions and the luminance information of the cell regions, a representative cell determination step of determining a representative cell region among the plurality of extracted cell regions based on at least one or more types of information among the information constituting the characteristic information, and a display control step of causing the display unit to display the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell.

Advantages of the Invention

[0009] According to the present invention, even when the cell image to be analyzed includes a large number of cells, the main cells (representative cells) can be extracted from the large number of cells and displayed together with the information regarding the representative cells.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment: Cell Image Processing System> Hereinafter, a cell image processing system according to the first embodiment of the present invention will be described.

[0013] FIG. 1 shows a functional block diagram of the cell image processing system according to the present embodiment. The cell image processing system 100 includes a cell data acquisition unit 110, a representative cell determination unit 120, a display control unit 130, and a storage unit 140. The cell data acquisition unit 110 includes an image acquisition unit 111, a cell region extraction unit 112, and a cell information acquisition unit 113. In FIG. 1, each part constituting the cell image processing system is shown as an integrated device, but a part of these parts may be configured by an external device (including those on the cloud).

[0014] (Cell Data Acquisition Unit) The cell data acquisition unit 110 acquires a cell image and a cell data group obtained by image analysis of the cell image. The cell data group is a data group related to the characteristic information of cell regions, such as the morphological information of the cell regions and the luminance information of the cells, obtained by image analysis for each cell region included in the cell image.

[0015] In this embodiment, the position information of the cell region includes at least any one of (1) a group of contour coordinates indicating the boundary of the cell region in the cell image, and (2) coordinates indicating the centroid position of the cell region in the cell image. The position information of the cell region according to this embodiment is not particularly limited as long as it indicates the position of the cell region. For example, it is the centroid coordinates of the cell region in the cell image and the information of the group of contour coordinates indicating the boundary of the cell region.

[0016] In this embodiment, the morphological information of the cell region includes at least any one of the area of the cell region, the diameter (radius) of the cell region, the circularity of the cell region, and the contour perimeter of the cell region.

[0017] Also, in this embodiment, the luminance information of the cell region includes at least any one of the average value of the luminance (value) of the cell region, the integrated value of the luminance in the cell region, and the standard deviation of the luminance in the cell region.

[0018] (Image acquisition unit) The image acquisition unit 111 acquires at least any one of a bright-field image of a cell, a fluorescence image of a cell, and a phase-contrast image of a cell (cell image data). The bright-field image is, for example, an image showing the morphology of a cell such as a bright-field image of a cell obtained at the in-focus position, or a bright-field image taken with a certain distance shifted in the optical axis direction from the in-focus position (hereinafter referred to as a defocused image). The defocused image has the characteristic that the contour part of the phase object including the cell is emphasized with high luminance or low luminance compared to the non-cell region (the region other than the cell region).

[0019] Also, the fluorescence image is, for example, an image capturing the autofluorescence of a cell by imaging the cell irradiated with an excitation light source of a specific wavelength (hereinafter referred to as an autofluorescence image). The phase-contrast image is obtained by photographing the cell with a phase-contrast microscope.

[0020] The cell image (cell image data) may be image data stored in advance in the storage unit 140, or may be acquired from an external storage area such as an HDD or cloud storage. Further, the cell image processing system 100 in the present embodiment can be connected to a cell image observation device (not shown) via a communication I / F, and an image captured by the cell image observation device can be acquired.

[0021] (Cell Region Extraction Unit) The cell region extraction unit 112 extracts a cell region from the cell image acquired by the image data acquisition unit 111. Here, extracting a cell region means extracting and acquiring the morphology of individual cells discretely present within the observation field of view of the cells, or extracting and acquiring the morphology of individual cells and the entire mass of cell groups existing in a closely packed mass state. Software used for extracting the cell region can be any known software without particular limitation.

[0022] (Cell Information Acquisition Unit) The cell information extraction unit 113 acquires characteristic information including the morphological information of the cell region and the luminance information of the cell region based on the cell image acquired by the image acquisition unit 111 and the cell region extracted by the cell region extraction unit 112.

[0023] (Representative Cell Determination Unit) The representative cell determination unit 120 determines a representative cell (main cell) constituting the cell data group based on the cell data acquired by the cell data acquisition unit 110. Specifically, a representative cell region is determined from among the plurality of extracted cell regions based on at least one or more types of information constituting the characteristic information. Hereinafter, a representative cell may be referred to as a representative cell.

[0024] The representative cell determination unit according to this embodiment may extract a plurality of subsets from a data group composed of characteristic information of a plurality of cell regions, and determine a representative cell region from each of the plurality of subsets. The following are examples of subset extraction methods. (1) Calculate the probability that the data of each cell region can occur based on at least one or more types of information constituting the characteristic information. (2) Extract, as subsets, data groups corresponding to each of a plurality of predetermined ranges of occurrence probabilities from the data group composed of the characteristic information of the plurality of cell regions. Other subset extraction methods include the following. (a) Calculate the distance of the data of each cell region from the center of gravity in the data of all the cell regions based on at least one or more types of information constituting the characteristic information. (b) Extract, as subsets, data groups corresponding to each of a plurality of predetermined ranges of distances from the data group composed of the characteristic information of the plurality of cell regions. Other subset extraction methods include the following. Additionally, there are the following methods. (i) Divide the data groups constituting the subsets into a plurality of clusters by unsupervised clustering. (ii) In each of the data groups corresponding to each cluster, determine the cell region corresponding to the data closest to the center of gravity position of the data group as the representative cell region. Details of the process of determining the representative cell will be described later.

[0025] (Display control unit) The display control unit 130 displays information regarding the representative cells determined by the representative cell determination unit 120. Specifically, the display control unit 130 causes the display unit to display the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell. Hereinafter, "the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell" may be referred to as information regarding the representative cell.

[0026] The image information of the cell region corresponding to the representative cell includes an image obtained by cutting out and enlarging a small region including at least the representative cell from the cell image acquired by the image acquisition unit. The characteristic information of the cell region corresponding to the representative cell includes measurement data regarding the representative cell obtained by measuring the cell image. The measurement data regarding the representative cell is characteristic information such as morphological information and luminance information corresponding to the representative cell determined by the representative cell determination unit 120 among the cell data group acquired by the cell data acquisition unit 110.

[0027] Also, when any one piece of information among the information regarding the representative cell is selected, the display control unit may control to display, on the display unit, the information other than the selected information in synchronization and emphasized together with the selected information. Specifically, when any one type of the information regarding the representative cell displayed on the display by a user operation is selected, the display control unit 130 may perform control of synchronous emphasized display for emphasizing and displaying information different from the selected information.

[0028] According to the present embodiment, a representative cell region (representative cell) can be determined from a cell image, and the image and characteristic information of the representative cell can be displayed on the display unit. Thereby, the user can easily grasp the information of the representative cell among a large number of cells in the cell image.

[0029] <Second Embodiment Cell Image Processing Method> FIG. 2 shows a flowchart of the cell image processing method according to the second embodiment of the present invention. The cell image processing method in the present embodiment includes a step of acquiring analysis data of cells (S201), a step of determining a representative cell (S201), and a step of displaying information related to the representative cell (S203).

[0030] S201 includes an image acquisition step of acquiring a cell image with a sample containing a plurality of cells as a subject, and a cell region extraction step of extracting a plurality of cell regions corresponding to the cells from the cell image. S202 includes a representative cell determination step of determining a representative cell region among the plurality of extracted cell regions based on at least one or more types of information constituting the characteristic information. Here, the characteristic information is information including the morphological information of the cell region and the luminance information of the cell region, which is acquired in the cell information acquisition step.

[0031] S203 includes a display control step of causing a display unit to display the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell.

[0032] Hereinafter, together with the description of the flow of FIG. 2, the details of each function of the cell image processing system 100 will be described.

[0033] (Acquisition of cell data) In step S201, the cell data acquisition unit 110 acquires a cell image and a cell image data group by performing image analysis (image processing) on the cell image.

[0034] FIG. 3 shows a flowchart for analyzing a cell image and acquiring cell data.

[0035] Hereinafter, an example of acquiring a defocused image and a self-fluorescence image and acquiring cell data by image analysis using FIG. 3 will be described.

[0036] (Acquisition of cell morphology image) In step S301, the image data acquisition unit 111 acquires a defocused image as a cell morphology image. The image acquired in this step may be any image having a feature in which the contrast of the cell contour portion is emphasized with respect to the non-cell region. For example, an image such as a phase contrast microscope image, an oblique illumination system image, or an image captured by an optical system in which the object side and the image side are telecentric (both-sided telecentric optical system) can be used. Note that the imaging position of the defocused image is preferably a position where the contrast of the cell contour portion is most emphasized.

[0037] (Acquisition of Fluorescent Image) In step S302, the image data acquisition unit 101 acquires an autofluorescent image of the cell as a fluorescent image. Here, the autofluorescent image is an image obtained by irradiating the cell with excitation light having a predetermined wavelength to excite an endogenous fluorescent substance in the cell, and imaging through a filter that cuts the excitation light source wavelength of the fluorescence.

[0038] The wavelength of the excitation light and the conditions of the cut filter may be selected according to the fluorescence characteristics of the endogenous component to be observed. For example, as a combination of the excitation light wavelength and the cut filter, when the endogenous component is NAD(P)H, excitation light of 360 nm and a long-pass filter of 430 nm are used. Also, in the case of flavins (such as FAD), excitation light of 450 nm and a long-pass filter of 530 nm are used. Examples of endogenous components used for autofluorescence observation include other autofluorescent molecules produced in cells, such as collagen, fibronectin, tryptophan, and folic acid, but other fluorescent endogenous components may also be used. Note that the autofluorescent image is an image including the same visual field as the cell morphology image acquired in step S301. Furthermore, it may be an image captured by a camera equipped with a two-dimensional image sensor in which two or more types of pixels having peak detection sensitivity in two or more different wavelength regions are regularly arranged.

[0039] For example, it is an RGB type color camera using a CMOS (Complementary Metal-Oxide Semiconductor) sensor as an optical sensor.

[0040] However, the optical sensor in this embodiment is not limited to a CMOS sensor, and a camera using a CCD sensor may also be used. Note that it is desirable to use an appropriate sensor or a camera using the same according to the wavelength of the light emitted by the cells to be observed.

[0041] FIG. 4 and FIG. 5 show examples of the defocused image and the autofluorescence image obtained in steps S301 and S302. Image 01 in FIG. 4 is the obtained defocused image, and image 02 in FIG. 5 is the autofluorescence image taken in the same field of view as image 01 by the excitation light of 360 nm and a long-pass filter of 430 nm. Note that both image 01 and image 02 are RGB color images taken by an RGB color camera using a CMOS sensor as the optical sensor.

[0042] 410 and 510 are images obtained by magnifying and displaying a partial small area of image 01 and image 02. The areas outlined in black in 410 are individual cell areas, and it is an image taken in a field of view containing approximately three thousand cells.

[0043] Note that, as the autofluorescence image, two or more autofluorescence images having different combinations of the wavelength of the excitation light and the characteristics of the cut filter on the observation side may be obtained. Further, as the fluorescence image, not limited to the autofluorescence image, a stained image obtained by observing a cell sample to which a staining reagent showing fluorescence characteristics at a specific wavelength is added may be obtained.

[0044] (Cell Region Extraction Process) In step S303, the cell region extraction unit 112 extracts the cell region from the cell morphology image obtained in step S301.

[0045] The extraction of the cell region may be performed by known image analysis methods. For example, in the case of the defocused image obtained as shown in FIG. 4, the RGB image obtained by the RGB color camera is converted into a single channel, and by performing threshold processing, it is possible to extract cell contour information indicating the boundaries of individual cell regions. Further, a mask image is created by filling the internal region of the extracted cell contour with black pixels and the external region with white pixels, and by assigning identifiers to each region by labeling the regions of isolated black pixels, individual cell regions in the cell image can be specified.

[0046] FIG. 6 shows an example of the mask image 01 created for the defocused image 01 of FIG. 4. 610 is an image obtained by magnifying a partial small region of the mask image 01. The cell regions 00001 to 00003 indicate a part of the cell regions labeled based on the mask image 01.

[0047] Although an example using a defocused image taken by an RGB color camera has been shown, this cell region extraction process is applicable to an image in which the contrast of the cell contour part is emphasized with respect to the non-cell region. For example, an image obtained by a phase contrast microscope or an image subjected to image processing so that the cell contour part is emphasized may be used.

[0048] (Luminance information extraction process) In step S704, the luminance information extraction unit 113 acquires luminance information corresponding to individual cell regions from the images acquired in steps S301 and S302 based on the cell regions acquired in step S303.

[0049] As an example, the case where an image taken by an RGB color camera is acquired in step S302 will be described.

[0050] First, the Bayer array data of the image is separated into an image having three channels corresponding to the RGB components. The Bayer array data is two-dimensional array data recording the luminance values received by the CMOS color sensor.

[0051] Subsequently, the luminance values of the Bayer array data are separated into three RGB components by separating them into the components of each color filter corresponding to the three wavelength regions. Note that one pixel of the Bayer array data has information on the luminance value of any one of the RGB components, but it may be complemented so that all pixels have information on the luminance values of the three RGB components. Also, preprocessing such as black level correction processing for subtracting the dark current value from each pixel value and filter processing for noise reduction may be performed.

[0052] Subsequently, the luminance information in each cell region acquired in step S303 is extracted. The luminance information is a scalar value indicating the respective luminance statistics of the RGB components in each cell region, and for example, the average value is calculated. Hereinafter, an example in which the average value is calculated will be described, but the statistic is not limited to this, and an integrated value, a standard deviation, or a plurality of types of statistics may be calculated.

[0053] In the case of a fluorescence image, luminance statistics may be calculated only for significant components among the RGB components according to the characteristics of the cut filter at the time of shooting. For example, in the case of an autofluorescence image taken with excitation light of 360 nm and a long-pass filter of 430 nm, luminance statistics are calculated only for the G component and the B component.

[0054] By repeatedly performing the calculation of the luminance statistics as described above for each image and each cell region, a cell data group that is a set of luminance information of the cell regions can be obtained.

[0055] Also, morphological information indicating the morphological characteristics of each cell region may be further obtained together with the luminance information. Specifically, scalar values such as area, diameter, circularity, or area envelope are calculated.

[0056] FIG. 7 shows an example of a cell data group obtained based on the images 01 and 02 shown in FIGS. 4 and 5 and the cell regions shown in FIG. 6. Table 700 shows, for each row corresponding to each cell region, the results of calculating the luminance information and the morphological information as described above in each column.

[0057] Here, among the three RGB components, the sum, ratio, or difference of two components may be calculated and used as the luminance information of the cell region. For example, calculate the average luminance value of the G component of Image 02 divided by the average luminance value of the B component of Image 02.

[0058] As described above, an example of obtaining a defocused image and a self-fluorescence image in step S201 and obtaining cell data through image processing has been explained.

[0059] In step S201, the cell data acquisition unit 110 acquires a cell data group having both morphological and functional characteristics of the cells and uses it for the process of step S202. Note that the position information of the extracted cells in the cell image is also associated with and held in the data of each cell region as shown in FIG. 7. Specifically, as the position information of the cells, the information of the contour coordinate group surrounding the cell region and the information of the centroid coordinates are held.

[0060] Note that an example of obtaining a cell data group using the defocused image 01 and the self-fluorescence image 02 taken in a specific field of view has been described, but it is not limited to this, and images taken in a plurality of fields of view may be obtained. In that case, step S201 is performed for each of the images taken in a plurality of fields of view or a plurality of cell samples to obtain a cell data group.

[0061] (Determination of representative cells) In step S202, the representative cell determination unit 120 determines the main cells constituting the cell data group based on the cell data group acquired in step S201.

[0062] First, estimate the probability density distribution based on the acquired cell data group. The probability density distribution is a distribution for estimating the probability of occurrence of a certain data. The probability density distribution is estimated by a method such as kernel density estimation, for example. FIG. 8 shows the result of estimating the probability density distribution by kernel density estimation using two types of information, the B component of image 02 and the G component of image 02, among the cell data group 700 shown in FIG. 7. 801 is a scatter plot with the B component of image 02 on the horizontal axis and the G component of image 02 on the vertical axis, plotting the data of each standardized cell region. 802 visualizes the probability density distribution estimated by kernel density estimation for the data group of 801. The darker the area, the higher the probability of occurrence of the data.

[0063] Subsequently, in the probability density distribution such as 802, set a plurality of regions with a predetermined occurrence probability. For example, set three regions with occurrence probabilities of 95 - 100%, 45 - 55%, and 0 - 5%. In 802, they are regions such as D01, D02, and D03 respectively. Here, region D01 is a region that occupies the majority of the cell data and contains a large amount of reference cell data in the cell group to be observed. Also, D02 contains a large number of cell groups that may have morphological or functional characteristics different from the cells contained in D01. D03 is a region that contains cells that are likely to be cells in an unexpected state. Unexpected cells include, for example, unexpected trait changes, contamination by other cells, exceptional states depending on the cell state (pathological state), and cell-like non-cellular substances generated during the observation process. Note that the method of setting the regions is not limited to the above. For example, two regions with occurrence probabilities of 95 - 100% and 0 - 5% may be used, and the range of the occurrence probability may also be adjusted.

[0064] Subsequently, data for representative cells is determined from each of the regions D01 to D03. Specifically, first, for D01, a cell data group corresponding to the region of D01 is extracted as a subset S01 of the cell data group 700. Then, several cell data are extracted from the subset S01 as a representative cell group G01 in the region D01. For example, four cell data may be extracted based on a clustering method without a teacher. It is preferable that the cell data included in the representative cell group G01 are cell data having features that are not similar to each other among the subsets. For example, it can be realized by using a clustering method without a teacher. More specifically, the subset S01 is divided into four clusters by a clustering method without a teacher such as K-Means, and the data closest to the centroid of the data in each cluster is determined as the representative cell data. The above procedure is similarly performed in the regions D02 and D03 to extract representative cell groups G02 and G03.

[0065] Through the above step S202, representative cell groups G01, G02, and G03 are extracted as the main cells constituting the cell data group.

[0066] In the description of step S202, an example of determining representative cells using two types of data among the cell data group as shown in FIG. 7 has been described. However, the present invention is not limited to this, and representative cell groups can be similarly determined even when one or more types of data are used. For example, only diameter data or all types of data may be used. When a data type that is particularly advantageous for expressing the properties of cells is known among a plurality of data types, it is preferable to configure such that the user selects that data type and determines representative cells using the selected data type. When it is not known, it is also effective to perform the processing of this step after converting each cell data of N dimensions composed of N scalar values into lower-dimensional data by a dimensionality reduction method such as principal component analysis. For example, in the case of the example in FIG. 7, each cell data is 10-dimensional data consisting of 10 items, and by applying principal component analysis to the cell data group 700, it is converted into 2-dimensional data. Then, representative cells are determined in this step using the converted 2-dimensional data.

[0067] (Display of information regarding representative cells) In step S203, the display control unit 130 displays the information regarding the representative cells extracted in step S202 on the display. As described above, the information regarding the representative cells is three types of information: a representative cell image, measurement data regarding the representative cells, and position information of the representative cells.

[0068] FIG. 10 shows an example of displaying information regarding representative cells. 1000 is a GUI screen displayed on the display. 1010 is the image acquired in step S201. For example, the defocused image 01 shown in FIG. 4 is displayed. On the defocused image 01, the position information of each representative cell extracted in step S202 is superimposed and displayed. The position information is, for example, based on the contour coordinate information of the representative cells, and a contour line as shown by the representative cell contour 02a in FIG. 10 is drawn on the defocused image 01.

[0069] In 1020, the measurement data regarding the representative cells acquired in step S201 is displayed. Here, it is desirable that the measurement data regarding the representative cells is highlighted after looking down at the overall image of the cell data group acquired in step S201. For example, after displaying a scatter diagram as shown by 801 in FIG. 8, the plot points corresponding to each representative cell are highlighted. The highlighting is, for example, a plot in a color different from the data other than the representative cells or a plot with a marker size larger than other plots. Note that the information displayed in 1020 is not limited to the two-dimensional scatter diagram as shown in FIG. 10, and may be a histogram, a box-and-whisker plot, or a map visualizing the probability density distribution estimated in step S202. Even in those cases, the data corresponding to each representative cell data is highlighted and drawn with a line or a point.

[0070] 1030 displays the representative cell images based on the position information of each representative cell extracted in step S202. FIG. 10 shows an image obtained by cutting out and enlarging each representative cell from the defocused image 01 of FIG. 4 at a predetermined size such that the center of gravity of the cell is at the center of the image.

[0071] The predetermined size may be determined by the user according to the size of the cells to be observed. As shown in the notations of the representative cell groups G01, G02, and G03 in FIG. 10, it is preferable to display in such a way that it can be determined which region of the probability density distribution in step S202 the representative cells belong to. Note that 1010 and 1030 are the defocused image 01 and the representative cell images cut out from the defocused image 01, respectively. When a plurality of images are acquired as described in step S201, it is possible to switch between these plurality of images. Specifically, the user selects the image to be displayed in the selection bar shown in 1040, and accordingly switches the image displayed in 1010 and each representative cell image in 1030. Not limited to this, a configuration may be adopted in which the image is switched by mouse or keyboard operation.

[0072] In addition, after the display control unit 130 displays information regarding the representative cells on the GUI screen, it performs synchronous highlighting control to highlight information regarding other types of representative cells in response to the selection of information regarding any type of representative cells by the user. For example, when the representative cell image 02a among the representative cell images displayed in 1030 as shown in FIG. 10 is selected by the user, the representative cell contour 02a and the representative cell data 02a corresponding to the representative cell image 02a are highlighted.

[0073] Above, from step S201 to step S203, a method for determining representative cells from the cell data group obtained based on the cell images and displaying information regarding the representative cells has been described. According to the present embodiment, the user can easily grasp the main representative cells constituting the cell data group.

[0074] The following describes a modification example in steps S201 to S203 of the second embodiment. Note that since the steps not mentioned in each modification example are the same as those in the second embodiment, the description thereof is omitted.

[0075] [Modification Example 1 of the Second Embodiment] In step S302, when the cell morphology image and the fluorescence image are not in the same visual field, alignment processing is performed on the cell morphology image and the fluorescence image data.

[0076] The alignment processing is performed by a method based on marking by an operator or a method by image processing.

[0077] In the method based on marking, the operator marks multiple points at the positions of the same cells in each of the cell morphology image and the autofluorescence image, and performs alignment by calculating an affine transformation matrix based on the coordinates of the marked points. Here, it is preferable that the number of marking points is 9 or more.

[0078] When performing by image processing, for example, alignment is performed using a mask image generated by threshold processing on the autofluorescence image and a non-cell region mask image generated from the defocused image in step S302. Here, for the alignment of the two mask images, methods such as template matching and phase-limited correlation method are used.

[0079] According to this modification example, even when the cell morphology image and the fluorescence image are not in the same visual field, it is possible to obtain cell data having both the morphological characteristics and the functional characteristics of the cells.

[0080] [Modification Example 2 of the Second Embodiment] In this embodiment, as a preferable example of obtaining cell data having both the morphological characteristics and the functional characteristics of the cells, an example of obtaining a morphological image and a fluorescence image respectively has been described. However, it is also applicable when the observation image of the target cell sample is only a morphological image or only a fluorescence image.

[0081] In the case of only the morphological image, the acquisition of the fluorescence image in step S302 of the second embodiment is not performed, and the process proceeds to step S303. The subsequent processing can be performed in the same manner as in the second embodiment.

[0082] In the case of only the fluorescence image, the acquisition of the morphological image in step S301 of the second embodiment is not performed, and the process proceeds to step S302. The extraction of the cell region in step S303 may use a known image processing method. For example, after converting the fluorescence image into a grayscale image, a mask image as shown in FIG. 6 is created and labeling processing is performed by threshold processing.

[0083] The threshold value in the threshold processing may be determined by a known method such as Otsu's binarization. The subsequent processing can be performed in the same manner as in the second embodiment described above.

[0084] [Modification Example 3 of the Second Embodiment] In step S304, although the method of extracting RGB components from the Bayer array data has been described, the RGB components may be converted into a different color space or color system and then each component may be extracted. For example, color spaces such as sRGB and AdobeRGB, or color systems such as Lab, XYZ, and HSV.

[0085] [Modification Example 4 of the Second Embodiment] In this embodiment, the autofluorescence information extraction method has been described by taking the RGB camera using a CMOS sensor as an example. However, an image obtained using a spectroscopic spectral camera such as a multi-band spectral camera or a hyperspectral camera having an optical sensor suitable for observation wavelengths other than the RGB camera may also be used. The optical sensor is, for example, of the CMOS, CCD, Ge type, InGaAs type, bolometer type, APD (avalanche photodiode array) type.

[0086] In that case, in step S304, the image is separated into images of a plurality of components corresponding to a plurality of wavelength filters of the camera.

[0087] [Modification Example 5 of the Second Embodiment] The estimation of the probability density distribution described in the second embodiment is effective when there is a dense cell data distribution in the data space composed of the types of data used for the estimation. On the other hand, in the case of a cell data group in which there is no dense cell data distribution, representative cells may be determined based on the distance from the center of gravity of the cell data group.

[0088] Specifically, a plurality of data distribution regions are set based on the distance from the center of gravity of the cell data group. FIG. 9 shows an example in which regions are set in a cell data group such as 801 in FIG. 8. For the center of gravity C01 of the cell data group, three regions D04, D05, and D06 corresponding to the ranges of distances 0.00 to 0.10, 0.45 to 0.55, and 0.95 to 1.00 are set. Note that the distance from the center of gravity is normalized so that the maximum value is 1.00.

[0089] The determination of the representative cells in each of the regions D04, D05, and D06 is the same as in the second embodiment.

[0090] Whether to use the estimation of the probability density distribution according to the second embodiment or the method according to this modification example may be determined using a known accuracy evaluation index in the estimation of the probability density distribution. Specifically, after calculating the occurrence probability of each data, the total value can be used as the numerical value of the accuracy evaluation index. The evaluation value is a numerical value from 0.0 to 1.0. For example, when the evaluation value is 0.8 or less, the determination is switched to the representative cells based on this modification example.

[0091] Note that a configuration may be adopted in which a scatter diagram such as 801 in FIG. 8 or a map showing the result of the probability density estimation of 802 is displayed, and the user can confirm the result and then switch which one to adopt.

[0092] [Modification Example 6 of the Second Embodiment] In the second embodiment, an example of displaying representative cell images in a list format was described with reference to FIG. 10. However, at least two or more types of information among the three types of information, i.e., the representative cell image, the position information of the representative cell, and the measurement data regarding the representative cell, may be displayed, and the format for displaying them is not limited to the example shown in FIG. 10.

[0093] For example, the display control unit may associate the image information of the cell region corresponding to a representative cell with the position of the representative cell region in the cell image and cause the display unit to display them. Specifically, it may be a format in which the representative cell image is mapped to the corresponding coordinates on the image displayed at 1010 or the position of the corresponding data in the scatter diagram at 1020 for display. FIG. 11 shows an example in which the representative cell image is mapped and displayed according to the corresponding coordinates on the image at 1010. After each representative cell image is displayed on the image 01 displayed at 1010, each representative cell is associated with the centroid position of the corresponding representative cell for display.

[0094] Also, instead of being displayed on the same GUI screen 1000, a configuration may be adopted in which each of 1010, 1020, and 1030 or any one of them is displayed as an independent separate GUI screen.

[0095] The above display formats shown in the second embodiment and this modification may be arbitrarily switched by the user. For example, it is possible to select an appropriate display format according to the type of the display device such as a desktop PC (Personal Computer), a laptop PC, a tablet PC, a smartphone, etc. and the resolution of the display.

[0096] [Third Embodiment] In the second embodiment, a method of determining a representative cell from a group of cell data obtained based on a cell image and displaying information regarding the representative cell was described.

[0097] In this embodiment, a method of performing cluster classification by unsupervised clustering on the obtained group of cell data and determining a representative cell in each cluster will be described.

[0098] According to this embodiment, even when a cell sample to be analyzed contains a large number of cell groups with different properties, it is possible to appropriately determine, as representative cells, the main cell data that constitutes the cell data group.

[0099] This embodiment is particularly effective when it can be assumed in advance that the target cell sample contains cell groups with different properties. For example, it is an image obtained by observing a cell sample in which two or more different cell types or cells subjected to two or more different reagent additions are co-cultured, and a cell sample in a state where a large number of live cells and dead cells are generated, respectively.

[0100] FIG. 12 shows a functional block diagram of this cell image processing system. The cell image processing system 1200 includes a cell data acquisition unit 1210, a classification unit 1220, a representative cell determination unit 1230, a display control unit 1240, and a storage unit 1250. Further, the cell data acquisition unit 1210 includes an image data acquisition unit 1211, a cell region extraction unit 1212, and a cell luminance information acquisition unit 1213.

[0101] The cell data acquisition unit 1210, the display control unit 1240, the cell data acquisition unit 1210, the image data acquisition unit 1211, the cell region extraction unit 1212, and the cell luminance information acquisition unit 1213 are the same as those in the first embodiment, and thus the description thereof is omitted.

[0102] (Classification Unit) The classification unit 1220 classifies the cell data group acquired by the cell data acquisition unit 1210 into a plurality of clusters by an unsupervised clustering method.

[0103] (Representative Cell Determination Unit) The representative cell determination unit 1230 determines, for each of the cell data groups of each cluster classified by the classification unit 1220, representative cell data that is the main cell data constituting the cluster.

[0104] FIG. 13 shows a flowchart of a cell image processing system according to a third embodiment. Hereinafter, details of each function of the cell image processing system 1200 will be described together with the description of the flowchart of FIG. 13.

[0105] Regarding step S1301, since it is the same as step S201 of the second embodiment, the description thereof will be omitted.

[0106] (Cluster Classification) In step S1302, the classification unit 1220 classifies the cell data group acquired in step S1301 into a plurality of clusters.

[0107] For example, by using an unsupervised clustering method based on K-Means with the number of clusters set to 2, the cell data group is classified into two clusters CL01 and CL02. Here, the number of clusters may be set to an arbitrary number by the user. For example, in the case of a cell image of a cell sample in which two different cell types are co-cultured, it is set to 2 clusters.

[0108] (Determination of Representative Cells) In step S1303, the representative cell determination unit 1230 determines representative cells in each cell data group belonging to each cluster based on the classification result of step S1302. The method for determining representative cells in each cluster is the same as that in the second embodiment. Based on the cell data group belonging to each cluster, the probability density distribution is estimated, and extraction of subsets and extraction of representative cell data are performed based on the occurrence probability of the data.

[0109] Note that in the classification result, depending on the cell sample, it is assumed that the number of cell data belonging to a specific classification result may be small. For example, it is the case where the number of cells belonging to a cluster is less than 100. In such a case, it is better not to perform the estimation of the probability density distribution in the second embodiment, the setting of predetermined regions such as D01 and D02 based thereon, and the extraction of subsets corresponding to each region. Instead, all the data belonging to the cluster may be treated as a data group equivalent to the subset in the second embodiment, and representative cell data may be determined.

[0110] For example, when a cell data group is classified into two clusters, cluster A and cluster B, and regions with occurrence probabilities of 95 - 100% and 45 - 55% are set, representative cell groups A01 and A02 of cluster A and representative cell groups B01 and B02 of cluster B are extracted as representative cells.

[0111] Also, as in the present embodiment, when classifying a cell data group into a plurality of clusters and then determining representative cells that make up each cluster, cell data existing near the boundary between different clusters is also important cell data for the user to understand the state of the cells. For example, when classifying the cell images of a cell sample in which two different cell types are co-cultured into two clusters, the data existing near the boundary of the clusters has similar characteristics among different cell types. Therefore, it is highly likely to be important data in the analysis of cell data by the user.

[0112] Therefore, in the present embodiment, data existing near the boundary of such clusters is also extracted as representative cell data. Specifically, for cell data belonging to an arbitrary cluster, the occurrence probability on the probability density distribution estimated based on the cell data group in other clusters is calculated, and a cell data group with an occurrence probability equal to or higher than a certain value is extracted as a subset. Then, representative cell data is determined from the subset in the same manner as in the second embodiment. For example, when classified into two clusters, cluster A and B, and probability density distributions DA and DB are estimated for each, the occurrence probability of the cell data belonging to cluster A on the probability density distribution DB is calculated. Then, a data group with an occurrence probability of 50% or more is extracted as a subset. Then, four representative cell data are extracted from the subset as a representative cell data group. Note that the extracted representative cell group is different from the representative cell group extracted by the method of the second embodiment, and an identifier such as representative cell group A03b is assigned so that it can be distinguished based on which cluster's probability density distribution it is extracted from. Similarly, based on the probability density distribution DA in cluster B, representative cell group B03a is extracted.

[0113] (Display of information about representative cells) In step S1304, the display control unit 1240 displays information about the representative cells in each cluster extracted in step S1303 on the display.

[0114] The representative cell images in each cluster may be displayed in a list format obtained by expanding 1030 in FIG. 10, with each row being the representative cell image in each cluster. Alternatively, as in 1450 of FIG. 14, a UI for selecting the cluster to be displayed may be provided, and the representative cell image displayed in 1030 may be switched according to the selected cluster.

[0115] As described above, according to the present embodiment, even when the cell sample to be analyzed contains a large number of cell groups with different properties, it is possible to appropriately determine the main cell data constituting the cell data group as representative cells.

[0116] As described above, the present embodiment is effective when it can be assumed in advance that the target cell sample may contain cell groups with different properties. However, when it is not certain whether the cell sample contains cell groups with different properties, a configuration in which the second and third embodiments are sequentially implemented is also effective. For example, after the user confirms the determination and display result of the representative cells according to the second embodiment as shown in FIG. 10, it is determined whether the cell sample contains a large number of cell groups with different properties. As a result, if it is determined that the cell sample contains such cell groups, the determination of representative cells including classification into each cluster in the third embodiment is performed.

[0117] <Program> The program according to the present embodiment is a program for causing a computer to execute the cell image processing method according to the present embodiment described so far.

[0118] FIG. 15 is a block diagram showing an example of the hardware configuration of an information processing system capable of executing the program according to the present embodiment.

[0119] The information processing system has the functions of a computer. For example, the information processing system may be integrated with a desktop PC (Personal Computer), a laptop PC, a tablet PC, a smartphone, or the like.

[0120] To realize the functions as a computer that performs calculations and storage, the information processing system includes a CPU (Central Processing Unit) 1501, a RAM (Random Access Memory) 1502, a ROM (Read Only Memory) 1503, and an HDD (Hard Disk Drive) 1004. The information processing system also includes a communication I / F (interface) 1605, a display device 1506, and an input device 1507. The CPU 1501, the RAM 1502, the ROM 1503, the HDD 1504, the communication I / F 1505, the display device 1506, and the input device 1507 are interconnected via a bus 1510. Note that the display device 1506 and the input device 1507 may be connected to the bus 1510 via a driving device (not shown) for driving these devices.

[0121] In FIG. 15, each part constituting the information processing system is illustrated as an integrated device, but a part of these functions may be constituted by an external device. For example, the display device 1506 and the input device 1507 may be external devices separate from the part constituting the functions of a computer including the CPU 1501 and the like.

[0122] The CPU 1501 performs a predetermined operation according to a program stored in the RAM 1502, the HDD 1504, etc., and also has a function of controlling each part of the information processing system. The RAM 1502 is composed of a volatile storage medium and provides a temporary memory area necessary for the operation of the CPU 1501. The ROM 1503 is composed of a non-volatile storage medium and stores necessary information such as a program used for the operation of the information processing system. The HDD 1504 is a storage device composed of a non-volatile storage medium and stores information regarding the number and position of individual independent partitions, fluorescence intensity, etc.

[0123] The communication I / F 1505 is a communication interface based on standards such as Wi-Fi (registered trademark) and 4G, and is a module for communicating with other devices. The display device 1506 is a liquid crystal display, an OLED (Organic Light Emitting Diode) display, etc., and is used for displaying videos, still images, characters, etc. The input device 1507 is a button, a touch panel, a keyboard, a pointing device, etc., and is used for a user to operate the information processing system. The display device 1506 and the input device 1507 may be integrally formed as a touch panel.

[0124] Note that the hardware configuration shown in FIG. 15 is an example, and devices other than these may be added, or some devices may not be provided. Also, some devices may be replaced with other devices having similar functions. Further, some functions may be provided by other devices via a network, and the functions constituting the present embodiment may be realized by being distributed among a plurality of devices. For example, the HDD 1504 may be replaced with an SSD (Solid State Drive) using a semiconductor element such as a flash memory, or may be replaced with cloud storage.

[0125] Note that the above disclosure includes the following configurations, methods, programs, and recording media.

[0126] (Configuration 1) An image acquisition unit that acquires a cell image with a sample containing a plurality of cells as a subject, A cell region extraction unit that extracts a plurality of cell regions corresponding to cells from the cell image, A cell information acquisition unit that acquires characteristic information including the morphological information of the cell region and the luminance information of the cell region, and a representative cell determination unit that determines a representative cell region among the plurality of extracted cell regions based on at least one or more types of information among the information constituting the characteristic information. A cell image processing system having a display control unit that causes an image display unit to display image information of a cell region corresponding to the representative cell, position information corresponding to the representative cell region, and characteristic information of the cell region corresponding to the representative cell.

[0127] (Configuration 2) The cell image processing system according to Configuration 1, wherein the position information includes at least any one of a group of contour coordinates indicating a boundary of a cell region in the cell image and coordinates indicating a center-of-gravity position of the cell region in the cell image.

[0128] (Configuration 3) The cell image processing system according to Configuration 1 or 2, wherein the morphological information of the cell region includes at least any one of an area of the cell region, a diameter of the cell region, a circularity of the cell region, and a contour perimeter of the cell region.

[0129] (Configuration 4) The cell image processing system according to any one of Configurations 1 to 3, wherein the luminance information of the cell region includes at least any one of an average value of luminance in the cell region, an integrated value of luminance in the cell region, and a standard deviation of luminance in the cell region.

[0130] (Configuration 5) The cell image processing system according to any one of Configurations 1 to 4, wherein the cell image includes at least any one of a fluorescence image, a bright-field image, and a phase-contrast image.

[0131] (Configuration 6) The cell image processing system according to any one of Configurations 1 to 5, wherein the representative cell determination unit extracts a plurality of subsets from a data group composed of characteristic information of the plurality of cell regions, and determines the representative cell region from each of the plurality of subsets.

[0132] (Configuration 7) The representative cell determination unit calculates the probability that data for each cell region can be generated based on at least one or more types of information among the information constituting the characteristic information, and extracts, as subsets, data groups corresponding to each of a plurality of predetermined ranges of generation probabilities from among the data group constituted by the characteristic information of the plurality of cell regions. The cell image processing system according to Configuration 6.

[0133] (Configuration 8) The representative cell determination unit calculates the distance of the data of each cell region from the center of gravity in the data of all the cell regions based on at least one or more types of information among the information constituting the characteristic information, and extracts, as subsets, data groups corresponding to each of a plurality of predetermined ranges of distances from among the data group constituted by the characteristic information of the plurality of cell regions. The cell image processing system according to Configuration 6.

[0134] (Configuration 9) The representative cell determination unit divides the data group constituting the subset into a plurality of clusters by unsupervised clustering, and in each data group corresponding to each cluster, determines the cell region corresponding to the data closest to the center of gravity position of the data group as the representative cell region. The cell image processing system according to any one of Configurations 6 to 8.

[0135] (Configuration 10) The cell image processing system further includes a classification unit that classifies the cell data group constituting the cell image into a plurality of clusters, and the representative cell determination unit determines the representative cell region in each of the plurality of clusters. The cell image processing system according to any one of Configurations 1 to 9.

[0136] (Configuration 11) When any one of the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell is selected, the display control unit controls to display, on the display unit, in synchronization and with emphasis, the information other than the selected information together with the selected information. The cell image processing system according to any one of Configurations 1 to 10.

[0137] (Configuration 12) The display control unit controls to display, on the display unit, by associating the image information of the cell region corresponding to the representative cell with the position of the representative cell region in the cell image. The cell image processing system according to any one of Configurations 1 to 10.

[0138] (Method) An image acquisition step of acquiring a cell image with a sample containing a plurality of cells as a subject; A cell region extraction step of extracting a plurality of cell regions corresponding to cells from the cell image; A cell information acquisition step of acquiring characteristic information including the morphological information of the cell region and the luminance information of the cell region; A representative cell determination step of determining a representative cell region among the plurality of extracted cell regions based on at least one or more types of information among the information constituting the characteristic information; A cell image processing method having a display control step of displaying, on the display unit, the image information of the cell region corresponding to the representative cell, the position information corresponding to the representative cell region, and the characteristic information of the cell region corresponding to the representative cell.

[0139] (Program) A program for causing a computer to execute the cell image processing method described in the method.

[0140] (Recording Medium) A recording medium having a program for causing a computer to execute the cell image processing method described in the method.

Claims

1. an image acquisition unit for acquiring a cell image of a sample including a plurality of cells; a cell region extraction unit that extracts a plurality of cell regions corresponding to cells from the cell image; A cell information acquisition unit that acquires characteristic information including morphological information of the cell region and brightness information of the cell region, and a representative cell determination unit that determines a representative cell region from among the extracted multiple cell regions based on at least one type of information constituting the characteristic information; A cell image processing system having a display control unit that displays image information of a cell region corresponding to the representative cell, position information corresponding to the representative cell region, and characteristic information of the cell region corresponding to the representative cell on a display unit.

2. The cell-image processing system according to claim 1 , wherein the position information includes at least one of a group of contour coordinates indicating a boundary of a cell region in the cell image and coordinates indicating a center of gravity position of the cell region in the cell image.

3. The cell-image processing system according to claim 1 , wherein the morphological information of the cell region includes at least one of an area of ​​the cell region, a diameter of the cell region, a circularity of the cell region, and a contour perimeter of the cell region.

4. The cell image processing system according to claim 1 , wherein the brightness information of the cell region includes at least one of an average brightness value in the cell region, an integrated brightness value in the cell region, and a standard deviation of the brightness in the cell region.

5. The cell image processing system according to claim 1 , wherein the cell image includes at least one of a fluorescent image, a bright field image, and a phase contrast image.

6. The cell image processing system according to claim 1 , wherein the representative cell determination unit extracts a plurality of subsets from a data group composed of characteristic information of the plurality of cell regions, and determines the representative cell region from each of the plurality of subsets.

7. The cell image processing system of claim 6, wherein the representative cell determination unit calculates the probability that data of each cell region may occur based on at least one type of information constituting the characteristic information, and extracts, as subsets, data groups corresponding to each of a plurality of predetermined occurrence probability ranges from a data group composed of characteristic information of the plurality of cell regions.

8. The cell image processing system of claim 6, wherein the representative cell determination unit calculates the distance of data for each cell region from the center of gravity of data for all of the cell regions based on at least one type of information constituting the characteristic information, and extracts, as subsets, data groups corresponding to each of a plurality of predetermined distance ranges from a data group composed of characteristic information for the plurality of cell regions.

9. The cell image processing system according to any one of claims 6 to 8, wherein the representative cell determination unit divides the data group constituting the subset into a plurality of clusters by unsupervised clustering, and determines, for each data group corresponding to each cluster, a cell region corresponding to data closest to the center of gravity of the data group as a representative cell region.

10. The cell image processing system according to claim 1 , further comprising a classification unit that classifies a group of cell data constituting the cell image into a plurality of clusters, wherein the representative cell determination unit determines a representative cell region in each of the plurality of clusters.

11. 2. The cell image processing system according to claim 1, wherein when any one of image information of a cell region corresponding to the representative cell, position information corresponding to the representative cell region, and characteristic information of the cell region corresponding to the representative cell is selected, the display control unit controls so that information other than the selected information is displayed on the display unit in a synchronized and highlighted manner together with the selected information.

12. The cell image processing system according to claim 1 , wherein the display control unit controls the display unit to display image information of a cell region corresponding to the representative cell in association with a position of the representative cell region in the cell image.

13. An image acquisition step of acquiring a cell image of a sample including a plurality of cells; a cell region extraction step of extracting a plurality of cell regions corresponding to cells from the cell image; a cell information acquiring step of acquiring characteristic information including morphological information of the cell region and brightness information of the cell region; a representative cell determination step of determining a representative cell region from among the extracted multiple cell regions based on at least one type of information constituting the characteristic information; A cell image processing method comprising a display control step of displaying image information of a cell region corresponding to the representative cell, position information corresponding to the representative cell region, and characteristic information of the cell region corresponding to the representative cell on a display unit.

14. A program for causing a computer to execute the cell image processing method according to claim 13.

15. A recording medium having a program for causing a computer to execute the cell image processing method according to claim 13.

Images