Time-lapse imaging system

Abstract

To provide an imaging apparatus and an information processing apparatus, which have a small apparatus size and can accurately image the entire culture region.SOLUTION: An imaging apparatus generates image data including an interference fringe image by imaging an observation object existing in a culture region of a culture container. The imaging apparatus includes: a plurality of light sources that irradiate the culture region with illumination light at different irradiation angles; and at least one imaging sensor that generates the image data by imaging the entire culture region each time each of the plurality of light sources irradiates the culture region with the illumination light. The imaging apparatus can be taken in and out of a culture chamber provided in an incubator.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an imaging device and an information processing device. [Background technology]

[0002] In recent years, the demand for infertility treatment has been increasing. Fertilized egg incubators are used to culture fertilized eggs that have undergone in vitro fertilization treatment. Fertilized eggs (embryos) cultured in incubators are then either transferred or frozen. An embryo is a fertilized egg that is in a dividing state.

[0003] Previously, to observe fertilized eggs during cultivation, it was necessary to remove the culture dish (also called a tray) containing the fertilized eggs from the incubator and then observe them under a microscope. However, removing the culture dish from the incubator caused stress to the fertilized eggs due to temperature changes, which was a problem.

[0004] Therefore, Patent Document 1 proposes an incubator that allows for observation of fertilized eggs being cultured without removing the culture dish. The incubator described in Patent Document 1 includes a culture section that holds a plurality of culture dishes in a culture environment, and an imaging section that is provided corresponding to the culture dishes held in the culture section. As described in Patent Document 1, an incubator that allows for observation of fertilized eggs while culturing them in the culture dishes without removing the culture dishes from the culture section is called a time-lapse incubator.

[0005] Furthermore, the imaging unit described in Patent Document 1 is an optical camera having a lens, and focus adjustment is performed by moving the lens along the optical axis. Human eggs are approximately spherical, with a diameter of approximately 100 to 150 μm. It is not known where in the egg the pronucleus and other components, which provide clues for determining whether an egg has been fertilized, are located. For this reason, the imaging unit described in Patent Document 1 captures multiple images with different focus positions.

[0006] Conventionally, microscopes such as phase-contrast microscopes have been used to observe cells, etc., as in the device described in Patent Document 1, but they required focusing when capturing an image of the object being observed. For this reason, lens-free digital holography, which does not require focusing when capturing an image of the object being observed, has recently come into use (see, for example, Patent Document 2).

[0007] In digital holography, an interference fringe image is captured by illuminating an object with coherent light such as laser light, and then the captured interference fringe image is reconstructed to generate a reconstructed image (a so-called tomographic image) at any focal position. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2018-093795 [Patent Document 2] Special Publication No. 2012-531584 Summary of the Invention [Problem to be solved by the invention]

[0009] In the cultivation of fertilized eggs, multiple fertilized eggs may be cultivated in one culture dish. For example, Patent Document 1 describes imaging four fertilized eggs simultaneously. However, the imaging unit described in Patent Document 1 is an optical camera with a lens for magnifying images, which poses a problem of increasing the size of the device when installed in an incubator. Furthermore, the device described in Patent Document 1 is unsuitable for small incubators for fertilized eggs because the size of the device increases if more optical cameras are added to image more fertilized eggs.

[0010] On the other hand, Patent Document 2 discloses imaging of fertilized eggs in an incubator using lens-free digital holography, but does not disclose imaging of multiple fertilized eggs simultaneously.

[0011] Furthermore, when simultaneously imaging a plurality of fertilized eggs cultured in a culture dish, there is a problem in that it is difficult to accurately image all of the fertilized eggs in the culture area.

[0012] As described above, when culturing cells such as fertilized eggs, there is a demand for an imaging device that is small in size and capable of capturing an image of the entire culture area with high accuracy.

[0013] The technology of the present disclosure aims to provide an imaging device and an information processing device that are small in size and capable of capturing an image of the entire culture area with high accuracy. [Means for solving the problem]

[0014] In order to achieve the above-mentioned object, the imaging device disclosed herein is an imaging device that generates image data including an interference fringe image by capturing an image of an object to be observed present in a culture region of a culture container, and is equipped with a plurality of light sources that irradiate the culture region with illumination light at different irradiation angles, and at least one imaging sensor that captures an image of the entire culture region each time each of the plurality of light sources irradiates the culture region with illumination light to generate image data, and is capable of being moved in and out of a culture chamber provided in an incubator.

[0015] The object to be observed is preferably a fertilized egg or a floating cell.

[0016] It is preferable that a plurality of observation objects are present in the culture region, and the imaging sensor simultaneously captures images of the plurality of observation objects.

[0017] It is preferable that the light source has a plurality of light emitting points, and the imaging sensor performs an imaging operation each time each of the light emitting points emits light, thereby generating a plurality of pieces of image data.

[0018] It is preferable to provide a plurality of image sensors, one for each observation object present in the culture area.

[0019] It is preferable that the image data is transmitted wirelessly via a communication unit.

[0020] The information processing device according to the present disclosure includes a processor that receives image data transmitted from the imaging device and performs reconstruction processing based on the received image data to generate a reconstructed image.

[0021] It is preferable that the processor identifies the observation area in which the object to be observed is located based on pre-imaging data obtained by the imaging device capturing an image of the entire culture area, extracts an image corresponding to the observation area from the image data, and performs reconstruction processing to generate a reconstructed image.

[0022] It is preferable that the processor identify an observation region where the object to be observed is located based on information about the culture vessel, extract an image corresponding to the observation region from the image data, and perform reconstruction processing to generate a reconstructed image.

[0023] Preferably, the processor generates a composite image by performing aperture synthesis processing on a plurality of reconstructed images corresponding to a plurality of light sources. [Effects of the Invention]

[0024] According to the technology of the present disclosure, it is possible to provide an imaging device and an information processing device that are small in size and capable of capturing an image of the entire culture area with high accuracy. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view illustrating an example of an imaging device. [Figure 2] FIG. 2 is a side view of the imaging device on which the culture vessel is placed. [Figure 3] FIG. 10 is a diagram showing an example of the relationship between a culture region and an imaging surface. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of an image sensor. [Figure 5] 10A and 10B are diagrams showing how an interference fringe image is generated by irradiating a fertilized egg with illumination light. [Figure 6] FIG. 2 is a diagram illustrating an example of image data generated by an imaging sensor. [Figure 7] FIG. 1 is a schematic diagram illustrating an example of the configuration of a time-lapse imaging system. [Figure 8] FIG. 2 is a block diagram showing an example of the internal configuration of an imaging device and an information processing device. [Figure 9] FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device. [Figure 10] FIG. 10 is a diagram illustrating an example of a reconstruction position. [Figure 11] 10 is a flowchart illustrating an example of the overall operation of the time-lapse imaging system. [Figure 12] 10A and 10B are schematic diagrams showing the configuration of a light emitting surface of a light source according to a modified example. [Figure 13] FIG. 10 is a side view of an imaging device according to a modified example. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of an information processing device according to a second embodiment. [Figure 15] FIG. 2 is a diagram showing an example of an observation target region. [Figure 16] FIG. 10 is a diagram illustrating an example of an image extraction process. [Figure 17] 10 is a flowchart showing an example of the overall operation of the time lapse imaging system according to the second embodiment. [Figure 18] FIG. 10 is a diagram showing a culture vessel according to a modified example. [Figure 19] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0027] [First embodiment] 1 shows an imaging device 10 according to a first embodiment. The imaging device 10 includes an illumination device 11, an imaging sensor 12, a support 13, a base 14, and a stage 15. The imaging device 10 performs so-called lens-free imaging, which captures an image of an object to be observed without using an optical lens.

[0028] The illumination device 11 is connected to one end of a substantially L-shaped support 13. The other end of the support 13 is connected to a base 14. The base 14 is flat and has a stage 15 provided in its approximate center. The stage 15 is provided with a recessed mounting portion 15A on which a culture container 20 for culturing a fertilized egg is placed. The support 13 supports the illumination device 11 so that the illumination device 11 faces the imaging surface 12A of the imaging sensor 12. The fertilized egg is an example of an "object to be observed" according to the technology of the present disclosure.

[0029] Hereinafter, the direction in which the illumination device 11 and the imaging surface 12A face each other will be referred to as the Z direction. A direction perpendicular to the Z direction will be referred to as the X direction. A direction perpendicular to the Z direction and the X direction will be referred to as the Y direction. The imaging surface 12A is perpendicular to the Z direction and parallel to the X and Y directions.

[0030] The imaging sensor 12 is configured, for example, by a monochrome CMOS (Complementary Metal Oxide Semiconductor) image sensor. The culture vessel 20 is placed on the imaging surface 12A of the imaging sensor 12. The culture vessel 20 is a shallow cylindrical vessel, also called a culture dish. The culture vessel 20 is transparent and transmits the illumination light 16 emitted from the illumination device 11. The diameter of the culture vessel 20 is approximately 30 to 60 mm. The thickness of the culture vessel 20 is approximately 10 to 20 mm.

[0031] A plurality of fertilized eggs 21 that have been subjected to in vitro fertilization treatment are seeded in the culture container 20. In vitro fertilization treatment includes intracytoplasmic sperm injection treatment performed under a microscope and regular in vitro fertilization treatment performed by combining eggs and sperm in a specified container. The method of fertilization of the fertilized eggs 21 to be cultured is not important. The fertilized eggs 21 are, for example, human fertilized eggs. The fertilized eggs 21 are approximately spherical and have a diameter of about 100 to 200 μm.

[0032] Each fertilized egg 21 is suspended in culture solution 22 dropped into culture vessel 20. Culture solution 22 is covered with oil 23 filled in culture vessel 20. Oil 23 prevents evaporation and pH changes of culture solution 22. Note that fertilized eggs 21 in a dividing state are also referred to as embryos. Fertilized eggs 21 in the present disclosure include embryos.

[0033] The imaging device 10 captures images of the multiple fertilized eggs 21 in a state where the culture vessel 20 is covered with a light-transmitting lid (not shown).

[0034] Fig. 2 is a side view of the imaging device 10 on which the culture vessel 20 is placed. As shown in Fig. 2, the illumination device 11 is composed of a base 17 and three light sources 18A, 18B, and 18C. The base 17 is connected to a support 13. The light sources 18A, 18B, and 18C are provided on the surface of the base 17 facing the stage 15.

[0035] The light sources 18A, 18B, and 18C are each configured with, for example, a laser diode, and emit illumination light 16 toward the stage 15. The light sources 18A, 18B, and 18C may each be configured with a combination of a light-emitting diode and a pinhole. The illumination light 16 is coherent light. The wavelength of the illumination light 16 is 640 nm, 780 nm, or the like. The illumination light 16 is radiant light.

[0036] Light sources 18A, 18B, and 18C each emit illumination light 16 at a different angle relative to imaging surface 12A of image sensor 12. Light source 18A is attached to base 17 at a position facing the center of imaging surface 12A, and emits illumination light 16 in a direction substantially perpendicular to imaging surface 12A.

[0037] Light source 18B is attached to a position offset in the +Y direction from the attachment position of light source 18A on base 17, and emits illumination light 16 obliquely toward imaging surface 12A. Light source 18C is attached to a position offset in the -Y direction from the attachment position of light source 18A on base 17, and emits illumination light 16 obliquely toward imaging surface 12A. Light source 18B and light source 18C are disposed at positions symmetrical in the Y direction with light source 18A as the center. Furthermore, light source 18B and light source 18C are each disposed on an inclined surface formed on base 17.

[0038] Furthermore, light sources 18A, 18B, and 18C each irradiate the entire culture area with illumination light 16. Here, the culture area is an area in the culture vessel 20 where a plurality of fertilized eggs 21 are cultured. For example, the culture area refers to the entire inner bottom surface 20A of the culture vessel 20 (see FIG. 3).

[0039] The imaging sensor 12 detects illumination light 16 emitted from each of the light sources 18A, 18B, and 18C and transmitted through the culture vessel 20. Specifically, the illumination light 16 enters the culture vessel 20 and is diffracted by the fertilized eggs 21, generating an interference fringe image that reflects the shape and internal structure of the fertilized eggs 21. The interference fringe image is also called a hologram image. The imaging sensor 12 simultaneously captures multiple interference fringe images generated by multiple fertilized eggs 21.

[0040] 3, the imaging sensor 12 has an imaging surface 12A larger in area than the culture region (i.e., the entire inner bottom surface 20A of the culture vessel 20) in order to capture an image of the culture region. For example, 12 fertilized eggs are cultured in the culture region of the culture vessel 20, and the imaging sensor 12 simultaneously captures 12 interference fringe images generated by illuminating light 16 incident on the 12 fertilized eggs.

[0041] 4 shows an example of the configuration of the image sensor 12. The image sensor 12 has a plurality of pixels 12B arranged on an imaging surface 12A. The pixels 12B are photoelectric conversion elements that photoelectrically convert incident light and output pixel signals according to the amount of incident light.

[0042] The pixels 12B are arranged at equal pitches along the X and Y directions. The arrangement of the pixels 12B is a so-called square arrangement. The X direction is a direction perpendicular to the Z direction. The Y direction is a direction perpendicular to the X and Z directions. The pixels 12B are arranged at a first arrangement pitch Δx in the X direction and at a second arrangement pitch Δy in the Y direction.

[0043] The image sensor 12 captures an image of light incident on the image pickup surface 12A, and outputs image data DT made up of pixel signals output from each of the pixels 12B.

[0044] 5 shows how an interference fringe image is generated by irradiating one fertilized egg 21 with illumination light 16. Part of the illumination light 16 incident on the culture vessel 20 is diffracted by the fertilized egg 21. That is, the illumination light 16 is split into diffracted light 30 that is diffracted by the fertilized egg 21 and transmitted light 31 that is not diffracted by the fertilized egg 21 and passes through the culture vessel 20. The transmitted light 31 is a spherical wave or a plane wave. The diffracted light 30 and transmitted light 31 pass through the bottom surface of the culture vessel 20 and are incident on the imaging surface 12A of the image sensor 12.

[0045] The diffracted light 30 and the transmitted light 31 interfere with each other to generate an interference fringe image 33. The interference fringe image 33 is composed of bright portions 36 and dark portions 38. In FIG. 5, the interference fringe image 33 is illustrated with the bright portions 36 and dark portions 38 each having a circular shape, but the shape of the interference fringe image 33 changes depending on the shape and internal structure of the fertilized egg 21. The imaging sensor 12 simultaneously captures a light image including multiple interference fringe images 33 formed on the imaging surface 12A, and outputs image data DT including the multiple interference fringe images 33.

[0046] 6 shows an example of image data DT generated by the imaging sensor 12. The image data DT includes interference fringe images 33 in a number corresponding to the number of fertilized eggs 21 cultured in the culture region of the culture vessel 20. The imaging sensor 12 captures an image of the culture region and outputs the image data DT each time illumination light 16 is irradiated onto the culture region from each of the light sources 18A, 18B, and 18C.

[0047] Fig. 7 shows an example of the configuration of a time-lapse imaging system. As shown in Fig. 7, the time-lapse imaging system 2 includes an imaging device 10, an incubator 40, and an information processing device 50. The incubator 40 is a multi-room incubator for fertilized eggs, and is also called an embryo culture device. The fertilized eggs 21 are cultured in the incubator 40 for a predetermined period (e.g., 7 days).

[0048] Unlike typical incubators for culturing cells other than fertilized eggs, the incubator 40 does not have one culture chamber, but has multiple culture chambers 41. This is because an imaging device 10 is housed in each culture chamber 41, allowing the fertilized eggs 21 to be managed individually to prevent them from being mistaken for other people's fertilized eggs 21. The culture chambers 41 are also referred to as culture chambers. Note that although the incubator 40 shown in FIG. 7 is provided with two culture chambers 41, the number of culture chambers 41 is not limited to this and can be changed as appropriate.

[0049] Each culture chamber 41 is provided with an openable / closable lid 42. The incubator 40 is provided with a switch 43 for opening and closing the lid 42 for each culture chamber 41. When the user operates the switch 43, the lid 42 is opened and closed by a drive mechanism (not shown). The lid 42 may also be configured to be opened and closed manually. When the lid 42 is closed, the culture chamber 41 is kept airtight.

[0050] A mixed gas of carbon dioxide (CO2) gas, nitrogen (N2) gas, and outside air is supplied from an external gas cylinder (not shown) to the culture chamber 41 via a HEPA filter (High Efficiency Particulate Air Filter). Heaters (not shown) are provided on the sides and bottom of the culture chamber 41. The culture chamber 41 maintains a constant culture environment by controlling the concentration, temperature, and humidity of the mixed gas to be constant.

[0051] The imaging device 10 is sized to be able to be inserted and removed into the culture chamber 41. As shown in FIG. 7 , one imaging device 10 is inserted into one culture chamber 41. That is, the lid 42 can be closed with the imaging device 10 on which the culture container 20 is placed inserted into the culture chamber 41. This allows images of the multiple fertilized eggs 21 to be taken by the imaging device 10 while culturing the multiple fertilized eggs 21 in the culture chamber 41, without removing the culture container 20 from the culture chamber 41.

[0052] The information processing device 50 is, for example, a desktop personal computer. A display 51, a keyboard 52, a mouse 53, and the like are connected to the information processing device 50. The keyboard 52 and the mouse 53 constitute an input device 54 for a user to input information. The input device 54 also includes a touch panel and the like.

[0053] The information processing device 50 wirelessly communicates with the imaging devices 10 housed in each of the culture chambers 41 to exchange data. The imaging devices 10 periodically (for example, every 5 to 15 minutes) capture images. The information processing device 50 periodically receives image data including an interference fringe image 33 (see FIG. 5) from the imaging devices 10, performs reconstruction processing based on the received image data, and displays the reconstructed image generated by the reconstruction processing. The reconstructed image is also called a tomographic image.

[0054] 8 shows an example of the internal configuration of the imaging device 10 and the information processing device 50. As shown in Fig. 8, the imaging device 10 includes, in addition to the lighting device 11 and the imaging sensor 12, a processor 60, a storage device 61, a communication unit 62, a power supply unit 63, and a battery 64, which are interconnected via a bus line 65.

[0055] The processor 60 is, for example, a field programmable gate array (FPGA) and controls the operation of each unit within the imaging device 10. The storage device 61 is, for example, a random access memory (RAM) or a flash memory. The storage device 61 stores image data generated by the imaging device 10 and various other data.

[0056] The communication unit 62 performs wireless communication with the information processing device 50. The processor 60 transmits image data to the information processing device 50 via the communication unit 62.

[0057] The battery 64 is a secondary battery such as a lithium polymer battery. The power supply unit 63 includes a power supply circuit and a charge control circuit. The power supply unit 63 supplies power supplied from the battery 64 to the processor 60 and the like. The power supply unit 63 also controls charging of the battery 64 using power supplied from an external source. The power supply unit 63 may be configured to be able to charge the battery 64 wirelessly.

[0058] The information processing device 50 includes a processor 55, a storage device 56, and a communication unit 57, which are interconnected via a bus line 58. The bus line 58 is also connected to the display 51 and input device 54 described above.

[0059] The processor 55 is configured by, for example, a CPU (Central Processing Unit), and realizes various functions by reading out an operating program 56A and various data stored in the storage device 56 and executing the processes.

[0060] The storage device 56 includes, for example, RAM, ROM (Read Only Memory), or a storage device. The RAM is, for example, a volatile memory used as a work area, etc. The ROM is, for example, a non-volatile memory such as a flash memory that stores the operating program 56A and various data. The storage device is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage stores an OS (Operating System), application programs, image data, various data, etc.

[0061] The communication unit 57 performs wireless communication with the communication unit 62 of the imaging device 10. The processor 55 receives image data transmitted from the imaging device 10 via the communication unit 57. The processor 55 also transmits a control signal for controlling imaging to the imaging device 10 via the communication unit 57.

[0062] The display 51 displays various screens. The information processing device 50 receives input of operation instructions from the input device 54 via the various screens.

[0063] Fig. 9 shows an example of the functional configuration of the information processing device 50. The functions of the information processing device 50 are realized by the processor 55 executing processing based on an operating program 56A. As shown in Fig. 9, the processor 55 includes an imaging control unit 70, an image data acquisition unit 71, a reconstruction processing unit 72, an aperture synthesis processing unit 73, and a display control unit 74.

[0064] The imaging control unit 70 controls the operation of the imaging device 10. Specifically, the imaging control unit 70 controls the operation of the lighting device 11 to generate illumination light 16 and the imaging operation of the imaging sensor 12 by transmitting a control signal to the imaging device 10. More specifically, the imaging control unit 70 causes the light sources 18A, 18B, and 18C included in the lighting device 11 to emit illumination light 16 in sequence, and causes the imaging sensor 12 to perform an imaging operation every time the illumination light 16 is emitted.

[0065] Hereinafter, the operation of generating illumination light 16 by lighting device 11 and the imaging operation of imaging sensor 12 will be collectively referred to as the imaging operation of imaging device 10. Based on an operation signal input from input device 54, imaging control unit 70 causes imaging device 10 to start the imaging operation.

[0066] The image data acquisition unit 71 acquires three pieces of image data DT generated and transmitted from the imaging device 10 after the imaging device 10 captures images of multiple fertilized eggs 21 in the culture container 20. The three pieces of image data DT correspond to light sources 18A, 18B, and 18C, and differ in the irradiation angle of the illumination light 16 relative to the imaging surface 12A of the imaging sensor 12. The image data acquisition unit 71 supplies the three pieces of image data DT acquired from the imaging device 10 to the reconstruction processing unit 72.

[0067] The reconstruction processing unit 72 generates three reconstructed images RP by performing calculations based on each of the three image data DT supplied from the image data acquisition unit 71. For example, as shown in FIG. 10, the reconstruction processing unit 72 generates a reconstructed image RP for a predetermined reconstruction position P, which is the height at which the fertilized egg 21 exists. The reconstruction position P is a position (so-called depth position) represented by a distance d from the imaging surface 12A of the imaging sensor 12 toward the illumination device 11. Note that the reconstruction position P may be set or changed by the user operating the input device 54.

[0068] The reconstruction processing unit 72 performs reconstruction processing based on the Fresnel transformation formulas expressed by the following formulas (1) to (3), for example.

[0069]

number

[0070]

number

[0071]

number

[0072] Here, I(x, y) represents image data. x represents the coordinate of pixel 12B of image sensor 12 in the X direction (see FIG. 4). y represents the coordinate of pixel 12B in the Y direction. Δx is the first array pitch mentioned above, and Δy is the second array pitch mentioned above (see FIG. 4). λ is the wavelength of illumination light 16.

[0073] As shown in equation (1), Γ(m,n) is a complex amplitude image obtained by Fresnel transforming the interference fringe image contained in the image data. Here, m = 1, 2, 3, . . . Nx-1 and n = 1, 2, 3, . . . Ny-1. Nx represents the number of pixels in the X direction of the image data. Ny represents the number of pixels in the Y direction of the image data.

[0074] As shown in equation (2), A0(m,n) is an intensity distribution image representing the intensity components of the complex amplitude image Γ(m,n). As shown in equation (3), φ0(m,n) is a phase distribution image representing the phase components of the complex amplitude image Γ(m,n).

[0075] The reconstruction processing unit 72 obtains a complex amplitude image Γ(m,n) based on equation (1), and obtains an intensity distribution image A0(m,n) or a phase distribution image φ0(m,n) by applying the obtained complex amplitude image Γ(m,n) to equation (2) or equation (3). The reconstruction processing unit 72 obtains either the intensity distribution image A0(m,n) or the phase distribution image φ0(m,n) and outputs it as a reconstructed image RP.

[0076] In this embodiment, the reconstruction processing unit 72 outputs the phase distribution image φ0(m,n) as the reconstructed image RP. The phase distribution image φ0(m,n) is an image that represents the refractive index distribution of the object being observed. Since the fertilized egg 21, which is the object being observed in this embodiment, is semi-transparent, most of the illumination light 16 is not absorbed by the fertilized egg 21 but is transmitted or diffracted, and therefore almost no image appears in the intensity distribution. For this reason, in this embodiment, it is preferable to use the phase distribution image φ0(m,n) as the reconstructed image RP.

[0077] The reconstruction processing unit 72 is not limited to a method using the Fresnel transform formula, and may perform reconstruction processing using a Fourier iterative phase retrieval method or the like.

[0078] The reconstruction processing unit 72 generates three reconstructed images RP by performing the above-mentioned reconstruction processing on each of the three image data DT supplied from the image data acquisition unit 71. The reconstruction processing unit 72 supplies the generated three reconstructed images RP to the aperture synthesis processing unit 73. The three reconstructed images RP correspond to the light sources 18A, 18B, and 18C, and differ in the irradiation angle of the illumination light 16 with respect to the imaging surface 12A of the image sensor 12.

[0079] The aperture synthesis processing unit 73 generates a composite image SP by performing aperture synthesis processing on the three reconstructed images RP supplied from the reconstruction processing unit 72. Specifically, the aperture synthesis processing unit 73 Fourier transforms each of the three reconstructed images RP to combine them in frequency space, and then generates a composite image SP by performing an inverse Fourier transform on the combined frequency data. Since the three reconstructed images RP are images of the object of observation at different irradiation angles of the illumination light 16, combining these images in frequency space captures high-frequency components of the object of observation, and a high-resolution composite image SP with high spatial resolution is obtained.

[0080] The display control unit 74 causes the display 51 to display the composite image SP generated by the aperture synthesis processing unit 73. In this embodiment, the display control unit 74 causes the display 51 to display the composite image SP corresponding to one reconstruction position P, but may also cause the display 51 to display multiple composite images SP corresponding to multiple reconstruction positions P.

[0081] Next, an example of the overall operation of the time-lapse imaging system 2 will be described with reference to the flowchart shown in Fig. 11. First, the user places the culture vessel 20 on the stage 15 of the imaging device 10, and then inserts the imaging device 10 into the culture chamber 41 of the incubator 40 (step S10). Note that it is sufficient to insert the imaging device 10 into at least one of the multiple culture chambers 41.

[0082] Next, the user closes the lid 42 of the culture chamber 41 and causes the incubator 40 to start culture (step S11). When the incubator 40 starts culture, the imaging device 10 captures images of the multiple fertilized eggs 21 cultured in the culture area within the culture container 20 under the control of the information processing device 50 (step S12). The imaging device 10 wirelessly transmits three pieces of image data DT generated by performing the imaging operation to the information processing device 50 (step S13).

[0083] The information processing device 50 receives the three image data DT transmitted from the imaging device 10 (step S14). The reconstruction processing unit 72 of the information processing device 50 performs reconstruction processing on each of the three image data DT to generate three reconstructed images RP corresponding to at least one reconstruction position P (step S15). The aperture synthesis processing unit 73 performs aperture synthesis processing based on the three reconstructed images RP generated by the reconstruction processing unit 72 to generate a composite image SP (step S16). The display control unit 74 causes the display 51 to display the composite image SP generated by the aperture synthesis processing unit 73 (step S17).

[0084] Next, the information processing device 50 determines whether the culture in the incubator 40 has finished (step S18). The culture is carried out for, for example, up to seven days from the start of the culture. The information processing device 50 determines whether the culture has finished based on, for example, the time elapsed since the start of the culture. If the information processing device 50 determines that the culture has not finished (step S18: NO), it determines whether a certain time (for example, 10 minutes) has passed since the previous image capture (step S19).

[0085] If the information processing device 50 determines that a certain time has passed since the previous image capture (step S19: YES), the process returns to step S12. The processes of steps S12 to S19 are repeatedly executed until the determination in step S18 is affirmative. After the information processing device 50 determines in step S18 that the culture in the incubator 40 has ended (step S18: YES), the user removes the imaging device 10 from the culture chamber 41 of the incubator 40 (step S20).

[0086] As described above, the imaging device 10 according to the technology of the present disclosure captures an interference fringe image by lens-free imaging without using an optical lens, and therefore has a small device size. Therefore, the imaging device 10 can be put in and taken out of the incubation chamber 41 of a small embryo incubator 40. The embryo incubator 40 is inexpensive because it does not have an optical camera or the like integrated into it.

[0087] Although time-lapse incubators with integrated optical cameras and the like are significantly more expensive than regular incubators, the imaging device 10 of the present disclosure is small and can be inserted into and removed from the culture chamber of a regular incubator. Therefore, the technology of the present disclosure allows an inexpensive regular incubator to be used as a time-lapse incubator.

[0088] The imaging device 10 according to the technology of the present disclosure is a lens-free imaging device that does not use optical lenses, and therefore has a wider imaging field of view than conventional microscopes that use optical lenses, and the imaging field of view is directly the imaging surface 12A of the imaging sensor 12. Therefore, the imaging device 10 can accurately capture an image of the entire culture vessel in a single imaging operation using the imaging sensor 12.

[0089] Furthermore, the imaging device 10 according to the technique of the present disclosure captures images of the fertilized egg 21 using three light sources 18A, 18B, and 18C that emit illumination light 16 at different angles relative to the imaging surface 12A of the imaging sensor 12, thereby obtaining information from multiple angles and therefore depth information of the fertilized egg 21. In the above embodiment, a high-resolution composite image SP is obtained by performing aperture synthesis processing that combines information from multiple angles to increase spatial resolution.

[0090] [Modification of the first embodiment] Next, a modification of the first embodiment will be described. In the first embodiment, the light sources 18A, 18B, and 18C are arranged along the Y direction, but the arrangement direction is not limited to this and they may be arranged along the X direction. Furthermore, in the first embodiment, the illumination device 11 has three light sources 18A, 18B, and 18C, but the number of light sources included in the illumination device 11 is not limited to three and may be two or more. The illumination device according to the technology disclosed herein may have multiple light sources that irradiate the culture region with illumination light at different irradiation angles. Note that "irradiating the culture region with illumination light at different irradiation angles" means that at least two of the illumination lights have different angles formed between a plane (e.g., the XZ plane in FIG. 2) including the vertical direction (Z-axis direction) and the central axis of the illumination light.

[0091] Each light source of the illumination device 11 may be a laser light source in which a plurality of light-emitting points (for example, 36 light-emitting points) are arranged in a two-dimensional array. A vertical cavity surface-emitting laser (VCSL) can be used as this laser light source. A single image data set DT including a high-resolution interference fringe image (so-called super-resolution interference fringe image) is obtained by synthesizing a plurality of image data obtained by the imaging sensor 12 performing an imaging operation while sequentially causing the plurality of light-emitting points included in one light source to emit light.

[0092] FIG. 12 illustrates the configuration of a light-emitting surface 19A of a light source 19 having a plurality of light-emitting points 19B. The light-emitting surface 19A is disposed in a position facing the image sensor 12. A plurality of light-emitting points 19B are arranged in a two-dimensional array on the light-emitting surface 19A. The arrangement pitch of the light-emitting points 19B is approximately 10 μm to 100 μm. Each of the light-emitting points 19B is selected in turn to emit illumination light 16. The light-emitting time interval between the plurality of light-emitting points 19B is several milliseconds.

[0093] The arrangement pitch of the light-emitting points 19B only needs to be different from the arrangement pitch of the pixels 12B, and does not necessarily have to be smaller than the arrangement pitch of the pixels 12B. For example, even if a light-emitting point 19B is located directly above an adjacent pixel 12B, the arrangement pitch of the light-emitting points 19B does not need to match the arrangement pitch of the pixels 12B. In this case, the illumination light 16 is irradiated at different positions on the pixels 12B. Therefore, when combining multiple image data, different pixels 12B that are located directly below each light-emitting point 19B and illuminated by the illumination light 16 are considered to be the same pixel, and alignment with an accuracy of one pixel or less can be performed, thereby generating a single image data DT that includes a super-resolution interference fringe image.

[0094] 12, the light-emitting points 19B are arranged in a 6×6 square, and 36 light-emitting points 19B are provided on the light-emitting surface 19A, but the number and arrangement pattern of the light-emitting points 19B are not limited to the number and arrangement pattern shown in FIG. 12. The greater the number of light-emitting points 19B, the higher the resolution of the interference fringe image can be, but the longer the calculation time for the synthesis process and reconstruction process. For this reason, it is preferable to optimize the number of light-emitting points 19B according to the required image quality and calculation time.

[0095] Furthermore, in the first embodiment, aperture synthesis processing is performed based on the multiple reconstructed images RP generated by the reconstruction processing unit 72, thereby generating a composite image SP containing depth information of the fertilized egg 21. The multiple reconstructed images RP generated by the reconstruction processing unit 72 are images in which the illumination light 16 is irradiated onto the fertilized egg 21 at different angles, so a three-dimensional image may be obtained by performing arithmetic processing using a filtered back projection method or the like used in radiation tomosynthesis imaging or the like.

[0096] Furthermore, in the first embodiment, the entire culture region is imaged by one imaging sensor 12, but the entire culture region may be imaged by multiple imaging sensors 12. For example, as shown in FIG. 13, one imaging sensor 12 may be provided for each fertilized egg 21 present in the culture region. In this case, each imaging sensor 12 outputs image data DT including one interference fringe image 33 (see FIG. 6). The reconstruction processing unit 72 generates a reconstructed image RP by performing reconstruction processing on each of the image data DT output from each imaging sensor 12. The aperture synthesis processing unit 73 generates a composite image SP using three reconstructed images RP for each imaging sensor 12. That is, the number of composite images SP generated is equal to the number of imaging sensors 12.

[0097] In this way, the imaging device according to the technique of the present disclosure may be one that includes at least one imaging sensor that captures an image of the entire culture region.

[0098] [Second embodiment] Next, a second embodiment will be described. The second embodiment is similar to the first embodiment except that the functions configured in the processor 55 of the information processing device 50 are different.

[0099] Fig. 14 shows the functional configuration of an information processing device 50 according to the second embodiment. The functions of the information processing device 50 are realized by a processor 55 executing processing based on an operating program 56A. As shown in Fig. 14, the processor 55 includes an imaging control unit 70, an image data acquisition unit 71, a reconstruction processing unit 72, an aperture synthesis processing unit 73, and a display control unit 74, as well as a region identification unit 75 and an image extraction unit 76.

[0100] In this embodiment, before executing an imaging operation for generating three pieces of image data DT, the imaging control unit 70 causes one of the three light sources 18A, 18B, and 18C to emit illumination light 16 and causes the imaging sensor 12 to perform an imaging operation (hereinafter referred to as a pre-imaging operation). For example, in the pre-imaging operation, the imaging control unit 70 causes the light source 18A located directly above the imaging sensor 12 to emit illumination light 16. In this embodiment, the imaging device 10 performs the pre-imaging operation, thereby outputting pre-imaging data PD.

[0101] The region specifying unit 75 specifies a region (hereinafter referred to as an observation target region R) in which the fertilized egg 21, which is the observation target, exists, based on the pre-imaging data PD output from the imaging device 10. The region specifying unit 75 supplies information including the multiple observation target regions R specified from within the culture vessel 20 based on the pre-imaging data PD to the image extraction unit 76.

[0102] As a result of the actual shooting operation using the three light sources 18A, 18B, and 18C, the image extraction unit 76 extracts images contained in the observation area R from each of the three image data DT acquired by the image data acquisition unit 71, and outputs them as extracted images CP.

[0103] In this embodiment, the reconstruction processing unit 72 generates a reconstructed image RP by performing reconstruction processing on each of the extracted images CP output from the image extraction unit 76. The aperture synthesis processing unit 73 generates a composite image SP using three reconstructed images RP for each observation target region R. That is, the number of composite images SP generated is equal to the number of observation target regions R.

[0104] The display control unit 74 causes the display 51 to display the multiple composite images SP generated by the aperture synthesis processing unit 73.

[0105] 15 shows an example of the observation target region R identified by the region identifying unit 75. The region identifying unit 75 identifies a region in which an observation target exists by performing image analysis on the pre-imaging data PD. For example, the region identifying unit 75 identifies a rectangular region containing one interference fringe image 33 as the observation target region R by performing template matching using a striped pattern as a template.

[0106] 16 shows an example of image extraction processing by the image extraction unit 76. The image extraction unit 76 extracts an image included in the observation region R from each of the image data DT, and sets it as an extracted image CP. The extracted image CP includes one interference fringe image 33.

[0107] Fig. 17 is a flowchart showing an example of the overall operation of the time lapse imaging system 2 according to the second embodiment. In the flowchart shown in Fig. 17, steps S30 to S32 are added to the flowchart shown in Fig. 11. The operation of steps S10 to S20 is basically the same as in the first embodiment.

[0108] In this embodiment, after step S11, the imaging device 10 performs a pre-imaging operation (step S30) under control of the information processing device 50. The pre-imaging data PD generated by the imaging device 10 is wirelessly transmitted to the information processing device 50.

[0109] The region specifying unit 75 of the information processing device 50 specifies the observation target region R based on the pre-imaging data PD (step S31). Information including the observation target region R specified by the region specifying unit 75 is supplied to the image extracting unit .

[0110] After step S31, the above-mentioned steps S12 to S14 are executed. After step S14, the image extraction unit 76 extracts an image included in the observation region R from each of the image data DT acquired in step S14, and supplies the extracted image CP to the reconstruction processing unit 72 (step S32). In step S15, the reconstruction processing unit 72 performs reconstruction processing on each of the extracted images CP output from the image extraction unit 76, thereby generating a reconstructed image RP.

[0111] As described above, in this embodiment, the reconstruction processing unit 72 does not perform reconstruction using the entire image data DT, but rather performs reconstruction using an extracted image CP with a small image size extracted from the image data DT, thereby speeding up the reconstruction process. Also, in this embodiment, reconstruction is not performed on areas of the image data DT that do not include the fertilized egg 21, which is the object of observation, so unnecessary processing can be eliminated. Also, in this embodiment, a composite image SP is generated for each fertilized egg 21, which is the object of observation, so it is possible to manage the images of each fertilized egg 21 individually.

[0112] [Modification of the second embodiment] Next, a modified example of the second embodiment will be described. In the second embodiment, the observation target region R is identified based on pre-imaging data PD obtained by imaging the observation target. Alternatively, the observation target region R may be identified based on information such as the size and / or shape of the incubation container 20.

[0113] Figure 18 shows a culture vessel 80 according to a modified example. Figure 18(A) is a plan view of the culture vessel 80. Figure 18(B) is a cross-sectional view of the culture vessel 80 taken along line AA shown in Figure 18(A). The culture vessel 80 is, for example, a culture dish dedicated to fertilized eggs.

[0114] A plurality of wells 81 are formed in the culture vessel 80. The wells 81 are concave depressions formed in the inner bottom surface of the culture vessel 80 to facilitate fixing the position of the fertilized egg 21. One fertilized egg 21 is seeded in each well 81, and the well 81 is filled with culture solution 22. The culture vessel 80 is filled with oil 23 so as to cover all of the wells 81.

[0115] When culturing is performed using the culture vessel 80, the fertilized egg 21, which is the object to be observed, is cultured in the well 81, and therefore the observation target region R corresponds to the formation region of the well 81. Therefore, the observation target region R can be identified by understanding the position and size of the well 81 in the culture vessel 80.

[0116] The size of the culture vessel 80, the number of wells 81, and the position and size of the wells 81 differ depending on the type of culture vessel 80. Therefore, in this modification, the region specifying unit 75 recognizes the type of culture vessel 80 based on the pre-imaging data PD, and specifies the observation target region R based on the recognized type of culture vessel 80.

[0117] 19, it is preferable to store container information 90 for each type of culture container 80 in the storage device 56. The container information 90 includes information indicating the position and size of the well 81. The region specifying unit 75 recognizes the type of culture container 80 based on the pre-image data PD, and acquires container information 90 corresponding to the recognized type of culture container 80 from the storage device 56. Then, the region specifying unit 75 identifies the observation target region R based on the acquired container information 90.

[0118] The region specifying unit 75 may specify the observation target region R based on input information (manufacturer information, etc.) input from the input device 54 or the like of the information processing device 50, without using the pre-imaging data PD. For example, the region specifying unit 75 may recognize the type of the culture vessel 80 based on the input information input from the input device 54 or the like, and obtain from the storage device 56 the vessel information 90 corresponding to the recognized type of the culture vessel 80.

[0119] Furthermore, the various modifications of the first embodiment described above can also be applied to the second embodiment. Furthermore, in the first and second embodiments, the object of observation is a fertilized egg, but the object of observation is not limited to a fertilized egg. The object of observation may be a floating cell other than a fertilized egg. A floating cell is a cell that floats in a culture medium. In addition to a fertilized egg, floating cells include CHO (Chinese Hamster Ovary) cells used for antibody production.

[0120] The time-lapse imaging systems 2 according to the first and second embodiments relate to a technique called lens-free imaging, in which the imaging device 10 does not include an optical lens. The technique of the present disclosure is applicable to digital holography in general (for example, when a reference light is used).

[0121] Various modifications are possible to the hardware configuration of the computer that constitutes the information processing device 50. For example, the information processing device 50 can be configured with multiple computers that are separated as hardware in order to improve processing capacity and reliability.

[0122] In this way, the hardware configuration of the computer of the information processing device 50 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 56A can be duplicated or stored in a distributed manner across multiple storage devices in order to ensure safety and reliability.

[0123] The following various processors can be used as the hardware structure of the processing units that perform various processes, such as the image data acquisition unit 71, the reconstruction processing unit 72, the aperture synthesis processing unit 73, the display control unit 74, the region identification unit 75, and the image extraction unit 76. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (operation program 56A) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD), a processor whose circuit configuration can be changed after manufacture, such as an FPGA, and an application specific integrated circuit (ASIC).

[0124] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0125] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0126] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0127] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction.

[0128] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0129] 2. Time-lapse imaging system 10. Imaging device 11 Lighting equipment 12 Image sensor 12A imaging surface 12B pixels 13 Posts 14 Foundation 15 stages 15A Placement section 16 Illumination 17 Foundation 18A,18B,18C light source 19 Light source 19A Light-emitting surface 19B Light-emitting point 20 Culture vessel 20A inner bottom 21 Fertilized egg 22 Culture solution 23 Oil 30 Diffracted Light 31 Transmitted light 33 Interference fringe image 36 Akabe 38 Dark part 40 Incubator 41 Culture room 42 Lid 43 Switch 50 Information processing equipment 51 Display 52 keyboards 53 Mouse 54 Input Devices 55 processors 56 Storage device 56A Operation Program 57 Communications Department 58 Bus Line 60 processors 61 Storage device 62 Communications Department 63 Power supply unit 64 Battery 65 Bus Line 70 Imaging control unit 71 Image data acquisition unit 72 Reconstruction processing unit 73 Aperture synthesis processing unit 74 Display control unit 75 Area identification part 76 Image Extraction Unit 80 Culture vessels 81 wells 90 Container Information CP extracted image DT image data P reconstruction position PD pre-imaging data R Observation area RP reconstructed image SP composite image

Claims

1. A time-lapse imaging system comprising: a culture chamber provided in an incubator; an imaging device that can be moved in and out of the culture chamber; and an information processing device that receives and processes image data transmitted from the imaging device, The imaging device is a plurality of light sources that irradiate the culture region of the culture vessel with illumination light at different irradiation angles; at least one image sensor that captures an image of the entire culture area each time each of the plurality of light sources irradiates the culture area with the illumination light, thereby generating the image data including an interference fringe image of an object to be observed present in the culture area; a communication unit that transmits the image data; Equipped with The information processing device includes: receiving the plurality of image data transmitted from the imaging device; Identifying an observation target region where the observation target exists based on pre-imaging data obtained by the imaging device imaging the entire culture region or information on the culture vessel, extracting an image corresponding to the observation region from each of the plurality of image data and performing a reconstruction process to generate a plurality of reconstructed images; generating a composite image by performing aperture synthesis processing on the plurality of reconstructed images corresponding to the plurality of light sources; Time-lapse imaging system.

2. The object to be observed is a fertilized egg or a floating cell. The time-lapse imaging system of claim 1 .

3. A plurality of the observation objects are present in the culture region, the imaging sensor simultaneously captures images of the plurality of observation objects; The time-lapse imaging system according to claim 1 or 2.

4. the light source has a plurality of light-emitting points; the imaging sensor performs an imaging operation each time each of the light-emitting points emits light, thereby generating a plurality of pieces of image data; The time-lapse imaging system according to any one of claims 1 to 3.

5. The imaging device includes a plurality of the imaging sensors, One imaging sensor is provided for each of the observation objects present in the culture area. The time-lapse imaging system according to any one of claims 1 to 4.

6. the communication unit wirelessly transmits the image data to the information processing device. The time-lapse imaging system according to any one of claims 1 to 5.

Images