Information processing system and method for control

Abstract

To improve a technique for detecting live cells or dead cells appearing in an image captured by imaging a plurality of cells.SOLUTION: An information processing system 10 includes: an imaging unit 11 having a first imaging element 111 for outputting a visible light image and a second imaging element 112 for outputting pixel information in which a luminance change is detected; and a control unit 14, wherein the imaging unit 11 starts imaging of a target region including one or more cells by using the first imaging element 111 and the second imaging element 112, and the control unit 14 acquires detection result information regarding a live cell or a dead cell included in the one or more cells on the basis of outputs of the first imaging element 111 and the second imaging element 112 and outputs the acquired detection result information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and a control method. [Background technology]

[0002] Conventionally, there are known techniques for detecting live or dead cells in an image of a plurality of cells. For example, Patent Document 1 discloses a diagnostic device that recognizes cell portions with a brightness higher than a first brightness set between the brightness of the background of the image and the brightness of live cells as live cells, and recognizes cell portions with a brightness lower than a second brightness set between the brightness of the background and the brightness of low activity cells as low activity cells or dead cells. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 07-082008 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, it is difficult to recognize cells in an image without using a staining solution such as trypan blue. In conventional techniques, dead cells are stained with a staining solution, and then the cells are imaged using a color television camera (visible light camera). However, some staining solutions are toxic, and the staining process is complicated. Therefore, there is room for improvement in the technology for detecting live or dead cells in an image of multiple cells.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for detecting live or dead cells in an image of a plurality of cells. [Means for solving the problem]

[0006] An information processing system according to an embodiment of the present disclosure includes: An information processing system comprising: an imaging unit having a first imaging element that outputs a visible light image and a second imaging element that outputs pixel information in which a change in luminance is detected; and a control unit, the imaging unit starts imaging a target area including one or more cells using the first imaging element and the second imaging element; The control unit obtaining detection result information regarding live cells or dead cells included in the one or more cells based on outputs of the first imaging element and the second imaging element; The acquired detection result information is output.

[0007] A control method according to an embodiment of the present disclosure includes: A control method for an information processing system including an imaging unit having a first imaging element that outputs a visible light image and a second imaging element that outputs pixel information in which a change in luminance is detected, and a control unit, The imaging unit starts imaging a target area including one or more cells using the first imaging element and the second imaging element; The control unit acquiring detection result information regarding live cells or dead cells included in the one or more cells based on outputs of the first imaging element and the second imaging element; and outputting the acquired detection result information. [Effects of the Invention]

[0008] According to one embodiment of the present disclosure, there is provided an improved technique for detecting live or dead cells in an image of a plurality of cells. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a schematic configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] 10 is a flowchart showing the operation of the information processing system. [Figure 3]FIG. 1 is a diagram visually explaining the operation of an information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described.

[0011] (Outline of the embodiment) An overview of an information processing system 10 according to an embodiment of the present disclosure will be described with reference to Fig. 1. Typically, the information processing system 10 is used to distinguish between live and dead cells in culture, but the use of the information processing system 10 is not limited to this example. As shown in Fig. 1, the information processing system 10 includes an imaging unit 11, an output unit 12, a storage unit 13, and a control unit 14. Details of each component of the information processing system 10 will be described later.

[0012] The imaging unit 11 also has a first imaging element 111 and a second imaging element 112. The first imaging element 111 is any imaging element that outputs a visible light image (for example, an RGB image). The second imaging element 112 is an event-based vision sensor (EVS) that outputs pixel information in which a change in luminance is detected.

[0013] First, an overview of this embodiment will be described, and details will be provided later. The information processing system 10 starts imaging a target region including one or more cells using a first imaging element 111 and a second imaging element 112. In the information processing system 10, the control unit 14 acquires detection result information regarding live cells or dead cells included in the one or more cells based on the outputs of the first imaging element 111 and the second imaging element 112. The information processing system 10 then outputs the acquired detection result information.

[0014] Generally, if dead cells are not stained, it is difficult to determine whether a cell in a visible light image is a live cell or a dead cell. In contrast, according to this embodiment, by combining the visible light image output by the first image sensor 111 and the pixel information output by the second image sensor 112, it is possible to accurately determine whether a cell in a visible light image is a live cell or a dead cell without staining.

[0015] That is, unlike dead cells, living cells behave by dividing or growing, and therefore the shape and position of the cells change, albeit slightly, over time. Here, the second imaging element 112 outputs pixel information in which a change in brightness (i.e., movement of the subject) is detected, as described above. Therefore, the pixel information output by the second imaging element 112 may contain only information about living cells, without including information about dead cells. In other words, the second imaging element 112 may function as a sensor that detects only living cells.

[0016] According to this embodiment, detection result information regarding live cells or dead cells is output based on the visible light image output by the first imaging element 111 and the pixel information output by the second imaging element 112. Therefore, according to this embodiment, it is possible to distinguish live cells from dead cells in a visible light image even without the step of staining dead cells, thereby improving the technology for detecting live cells or dead cells in an image obtained by capturing a plurality of cells.

[0017] Next, each component of the information processing system 10 will be described in detail with reference to FIG.

[0018] (Configuration of information processing system) As described above, the information processing system 10 includes the imaging unit 11, the output unit 12, the storage unit 13, and the control unit 14.

[0019] The imaging unit 11 is a camera including a first imaging element 111 and a second imaging element 112. The imaging unit 11 may include an optical system that forms an image of a subject on each of the first imaging element 111 and the second imaging element 112. The imaging unit 11 may include a light source that irradiates illumination light when capturing an image.

[0020] The first imaging element 111 is any imaging element that outputs a visible light image (for example, an RGB image) as described above. For example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used as the first imaging element 111.

[0021] As described above, the second imaging element 112 is an event-based vision sensor (EVS) that outputs pixel information in which a change in luminance is detected. In this embodiment, the "pixel information" is described as a binary image (hereinafter also referred to as an "EVS image") in which a pixel signal in which a change in luminance is detected is set to "1" and a pixel signal in which a change in luminance is not detected is set to "0." However, the pixel information is not limited to this example. For example, the pixel information may be coordinate information of each pixel in which a change in luminance is detected.

[0022] For ease of explanation, this embodiment will be described assuming that the first imaging element 111 and the second imaging element 112 have the same number of pixels, aspect ratio, and frame rate. Also, the visible light image output by the first imaging element 111 and the EVS image output by the second imaging element 112 have a one-to-one correspondence between the coordinates of each pixel, and two pixels with the same coordinates will be described assuming that they are capturing the same location on the subject.

[0023] The output unit 12 includes one or more output devices that output information. The output devices are, for example, but not limited to, a display or a speaker. Alternatively, the output unit 12 may include an interface for connecting an external output device.

[0024] The storage unit 13 includes one or more memories. The memories may be, for example, semiconductor memories, magnetic memories, optical memories, or the like, but are not limited to these. Each memory included in the storage unit 13 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 13 stores any information used in the operation of the information processing system 10. For example, the storage unit 13 may store system programs, application programs, embedded software, and the like.

[0025] For example, the storage unit 14 stores a first learning model and a second learning model.

[0026] The first learning model is a machine learning model that is trained to output a first output signal indicating information about cells included in the visible light image (e.g., the number of cells, the coordinates of each cell, the shape of each cell, etc.) in response to a first input signal including the visible light image output by the first imaging element 111. The first learning model is trained by, for example, supervised learning, but is not limited to this example and may be trained by any machine learning algorithm, such as deep learning.

[0027] The second learning model is a machine learning model trained to output a second output signal indicating information about living cells included in the pixel information (e.g., the number of living cells, the coordinates and area of ​​each living cell in the EVS image, and the outline of each living cell, etc.) in response to a second input signal including pixel information (e.g., an EVS image) output by the second imaging element 112. The second learning model is trained, for example, by supervised learning, but is not limited to this example and may be trained by any machine learning algorithm, such as deep learning.

[0028] The control unit 14 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process, but is not limited to these. The programmable circuit may be, for example, a FPGA (Field-Programmable Gate Array), but is not limited to these. The dedicated circuit may be, for example, an ASIC (Application Specific Integrated Circuit), but is not limited to these. Hereinafter, the processor, programmable circuit, and dedicated circuit will be referred to as "processor, etc." without any particular distinction being made. The control unit 14 controls the operation of the entire information processing system 10.

[0029] For ease of explanation, the present embodiment will be described assuming that the above-described components of the information processing system 10 are arranged in one device. However, the components of the information processing system 10 may be distributed across multiple devices that can communicate with each other. When the above-described components are distributed across multiple devices, the control unit 14 includes multiple processors, etc., and at least one processor, etc. is arranged in each of the multiple devices. In one example, the imaging unit 11 may be arranged in a single independent device (e.g., an edge device).

[0030] (Operation flow of information processing system) The operation flow of the information processing system 10 according to this embodiment will be described with reference to FIGS.

[0031] Step S100: The imaging unit 11 uses the first imaging element 111 and the second imaging element 112 to start imaging a target area including one or more cells.

[0032] Typically, a user culturing cells positions a container containing the cells being cultured so that the imaging unit 11 can capture the image, and then performs an operation to cause the information processing system 10 to start capturing the image. In response to this operation, the imaging unit 11 starts capturing the image of the positioned cells. In this embodiment, the region captured by the imaging unit 11 is referred to as a target region.

[0033] Step S101: The control unit 14 acquires the outputs of the first imaging element 111 and the second imaging element 112 from the imaging unit 11.

[0034] Specifically, the control unit 14 acquires the visible light image output by the first imaging element 111 and pixel information (e.g., an EVS image) output by the second imaging element 112. The visible light image and pixel information are acquired at a predetermined frame rate (e.g., 60 FPS) while the target area is being imaged. For example, in FIG. 3, the visible light image contains 30 living cells, and the EVS image contains 19 living cells.

[0035] Step S102: Based on the outputs of the first and second image sensors 111 and 112, the control unit 14 obtains detection result information regarding live or dead cells contained in one or more cells in the target region.

[0036] The "detection result information" may include any information regarding at least one of live cells and dead cells contained in one or more cells in the target region.

[0037] In one example, the detection result information may include a ratio (e.g., cell viability) of the number of live cells (hereinafter also referred to as the "second number B") to the number of one or more cells (hereinafter also referred to as the "first number A") in the target region, the first number A, the second number B, the area occupied by live cells in the target region, or a combination thereof. Specifically, the control unit 14 may input a first input signal including a visible light image output by the first imaging element 111 to a first learning model, and determine the number of cells indicated in a first output signal of the first learning model for the visible light image as the first number A. The control unit 14 may also input a second input signal including pixel information output by the second imaging element 112 to a second learning model, and determine the number of live cells indicated in a second output signal of the second learning model for the pixel information as the second number B. The control unit 14 may also calculate the sum of the areas of the live cells in the EVS image indicated in the second output signal. The control unit 14 may calculate the ratio of the total value to the area of ​​the EVS image as the occupation area ratio of live cells to the target region.

[0038] The detection result information may also include the ratio of the number of dead cells to the first number A, the first number, the number of dead cells, the area occupied by dead cells relative to the target region, or a combination thereof. The number of dead cells and the area occupied by dead cells relative to the target region can be determined in the same manner as the second number B and the area occupied by live cells relative to the target region described above.

[0039] The detection result information may also include an image in which a marker indicating each live cell or each dead cell among one or more cells in the target region is superimposed on the visible light image output by the first imaging element 111. Specifically, the control unit 14 may input a second input signal including pixel information output by the second imaging element 112 to a second learning model and identify coordinates of each live cell in the EVS image indicated in the second output signal of the second learning model corresponding to the pixel information. Here, as described above, the coordinates of each pixel in the visible light image output by the first imaging element 111 and the EVS image output by the second imaging element 112 have a one-to-one correspondence. Therefore, the identified coordinates in the EVS image can be treated as being equal to the coordinates in the visible light image. The control unit 14 may identify, among one or more cells included in the visible light image output by the first imaging element 111, each cell indicated by the identified coordinates as a live cell and each other cell as a dead cell. The control unit 14 may then generate an image in which a marker indicating each live cell or dead cell is superimposed on the visible light image. For example, the detection result information shown in FIG. 3 is an image in which markers are superimposed on 19 live cells out of 30 cells on the visible light image. Here, each marker superimposed on a live cell (or dead cell) may include a serial number. Alternatively, the control unit 14 may generate an image in which a first marker is superimposed on each live cell in the visible light image, and a second marker that is distinguishable from the first marker is superimposed on each dead cell.

[0040] Step S103: The control unit 14 outputs, via the output unit 12, the detection result information acquired in step S102.

[0041] Specifically, the control unit 14 outputs the detection result information via the display and / or speaker of the output unit 12.

[0042] Step S104: The control unit 14 determines whether or not a predetermined alert condition is satisfied based on the detection result information acquired in step S102. If it is determined that the alert condition is satisfied (step S104-Yes), the process proceeds to step S105. On the other hand, if it is determined that the alert condition is not satisfied (step S104-No), the imaging unit 11 ends imaging of the target area, and the process ends.

[0043] The "alert condition" may be preset or may be arbitrarily set by the user. For example, the alert condition may be a condition that the ratio of the second number B to the first number A (e.g., cell viability), the second number B, or the area occupied by live cells in the target region is less than a predetermined threshold.

[0044] Step S105: If it is determined in step S104 that the alert condition is satisfied (step S104-Yes), the control unit 14 outputs alert information via the output unit 12.

[0045] The "alert information" may include any information that notifies the user that the above-described alert condition is satisfied. For example, the control unit 14 may output predetermined image information or audio information as the alert information via the output unit 12. The alert information may also include a message according to the above-described alert condition. For example, if an alert condition is set that the ratio of the second number B to the first number A (e.g., cell viability), the second number B, or the area occupied by live cells in the target region is less than a predetermined threshold, the control unit 14 may output a message via the output unit 12 indicating that the cell viability, the number of live cells, or the area occupied by live cells in the target region is less than the threshold.

[0046] As described above, the information processing system 10 according to this embodiment starts imaging a target region including one or more cells using the first imaging element 111 and the second imaging element 112. In the information processing system 10, the control unit 14 acquires detection result information regarding live or dead cells included in the one or more cells based on the outputs of the first imaging element 111 and the second imaging element 112. The information processing system 10 then outputs the acquired detection result information.

[0047] With this configuration, detection result information regarding live cells or dead cells is output based on the visible light image output by the first imaging element 111 and the pixel information output by the second imaging element 112. Therefore, as described above, this embodiment improves the technology for detecting live cells or dead cells in an image obtained by capturing a plurality of cells, in that it becomes possible to distinguish between live cells and dead cells in a visible light image even without the step of staining the dead cells.

[0048] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0049] In the above-described embodiment, an example has been described in which, in step S102, the control unit 14 acquires information about cells contained in the visible light image and information about live cells contained in the pixel information by using a first learning model and a second learning model based on the outputs of the first imaging element 111 and the second imaging element 112. However, the acquisition of information about cells contained in the visible light image and information about live cells contained in the pixel information is not limited to this example in which learning models are used, and any method can be adopted. For example, the control unit 14 may acquire information about cells contained in the visible light image by applying any image recognition process, such as pattern matching, to the visible light image output by the first imaging element 111. Similarly, the control unit 14 may acquire information about live cells contained in the pixel information (e.g., an EVS image) output by the second imaging element 112 by applying any image recognition process.

[0050] In the above-described embodiment, an example was described in which the second output signal of the second learning model indicates information about living cells contained in the input pixel information, such as the number of living cells, the coordinates and area of ​​each living cell in the EVS image, and the outline of each living cell. However, the information about living cells indicated by the second output signal is not limited to these examples. For example, the second learning model may be a machine learning model trained to output, in response to a second input signal including pixel information (e.g., an EVS image) output by the second imaging element 112, a second output signal indicating the state of each living cell, in addition to the number of living cells, the coordinates and area of ​​each living cell in the EVS image, and the outline of each living cell, as information about living cells contained in the pixel information. The "state of living cells" may indicate any state corresponding to the behavior of living cells, such as "stationary," "growing," or "dividing."

[0051] In such a case, when the control unit 14 generates an image in which a marker indicating each living cell is superimposed on the visible light image output by the first imaging element 111 as detection result information, the marker may have different visibility depending on the state of the corresponding living cell. For example, a marker of a first color or shape may be superimposed on a living cell in a "stationary" state, a marker of a second color or shape may be superimposed on a living cell in a "growing" state, and a marker of a third color or shape may be superimposed on a living cell in a "dividing" state.

[0052] In such a case, the detection result information acquired by the control unit 14 may include the ratio of the number of live cells in each state (hereinafter also referred to as the "third number C") to the number of one or more cells in the target area (first number A), the third number C, or a combination thereof.

[0053] Also, an embodiment is possible in which, for example, a general-purpose computer functions as the information processing system 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing system 10 according to the above-described embodiment is stored in the memory of the computer, and the program is read and executed by a processor or the like of the computer. Therefore, the present disclosure can also be realized as a program executable by a processor or the like, or a non-transitory computer-readable medium storing the program. [Explanation of symbols]

[0054] 10 Information Processing Systems 11 Imaging unit 111 First image sensor 112 Second imaging element 12 Output section 13 Storage section 14 Control Unit

Claims

1. An information processing system including an imaging unit having a first imaging element that outputs a visible light image and a second imaging element that outputs pixel information in which a change in luminance is detected, and a control unit, the imaging unit starts imaging a target region including one or more cells using the first imaging element and the second imaging element; The control unit obtaining detection result information regarding live cells or dead cells included in the one or more cells based on outputs of the first imaging element and the second imaging element; outputting the acquired detection result information; Information processing system.

2. 2. The information processing system according to claim 1, An information processing system, wherein the detection result information includes an image in which a marker indicating each live cell or each dead cell among the one or more cells is superimposed on a visible light image that is the output of the first imaging element.

3. 2. The information processing system according to claim 1, An information processing system, wherein the detection result information includes a ratio of a second number of live cells to a first number of the one or more cells, the first number, the second number, the area occupied by live cells in the target region, or a combination thereof.

4. 4. The information processing system according to claim 3, The control unit outputs alert information when the ratio, the second number, or the occupied area rate included in the detection result information is less than a threshold value.

5. 4. The information processing system according to claim 3, Further comprising a storage unit that stores the first learning model and the second learning model; The control unit inputting a visible light image output from the first imaging element into the first learning model; identifying the first number based on a first output signal of the first learning model for the input visible light image; inputting pixel information that is the output of the second imaging element into the second learning model; An information processing system that identifies the second number based on a second output signal of the second learning model for the input pixel information.

6. 6. The information processing system according to claim 5, The control unit further identifies a state of each living cell among the one or more cells based on the second output signal; the detection result information includes an image in which a marker indicating each living cell among the one or more cells is superimposed on a visible light image that is the output of the first imaging element, and The marker has different visibility depending on the state of the corresponding living cell.

7. 6. The information processing system according to claim 5, The control unit further identifies a state of each living cell among the one or more cells based on the second output signal; An information processing system, wherein the detection result information further includes a ratio of a third number of live cells in each state to the first number of the one or more cells, the third number, or a combination thereof.

8. A control method for an information processing system including an imaging unit having a first imaging element that outputs a visible light image and a second imaging element that outputs pixel information in which a change in luminance is detected, and a control unit, the imaging unit starts imaging a target region including one or more cells using the first imaging element and the second imaging element; The control unit acquiring detection result information regarding live cells or dead cells included in the one or more cells based on outputs of the first imaging element and the second imaging element; outputting the acquired detection result information; A control method comprising:

Images