Information processing device, information processing method, and program

Abstract

An information processing device for processing image data acquired from an image capturing device that includes a light source and an image capturing sensor and generates image data by capturing an interference fringe image resulting from irradiation of an object to be observed with illumination light, said information processing device including a processor, wherein the processor extracts a feature amount from the image data and determines, on the basis of the feature amount, the quality of the interference fringe image contained in the image data.

Description

[Technical Field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, microscopes such as phase-contrast microscopes have been used to observe cells, but they required focusing when capturing an image of the object being observed. For this reason, lens-free digital holography, which does not require focusing when capturing an image of the object being observed, has recently come into use (see, for example, International Publication No. 2018 / 158947).

[0003] In digital holography, an interference fringe image is captured by illuminating an object with coherent light such as laser light, and then the captured interference fringe image is reconstructed to generate a reconstructed image (a so-called tomographic image) at any focal position. Summary of the Invention [Problem to be solved by the invention]

[0004] However, digital holography uses interference fringe images generated by irradiating the object being observed with light such as laser light. Therefore, if there is a foreign object such as an air bubble between the light source and the object being observed, the interference fringe image may not be generated due to the influence of the refractive index of the foreign object, and a reconstructed image representing the object being observed may not be obtained.

[0005] The reconstruction process takes time, so if it is discovered after the reconstruction process that an interference fringe image has not been obtained, photographing the object being observed again will waste time and reduce observation efficiency.

[0006] The technology of the present disclosure aims to provide an information processing device, an information processing method, and a program that enable improvement in observation efficiency. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device disclosed herein is an information processing device that acquires and processes image data from an imaging device that has a light source and an imaging sensor and captures an interference fringe image generated by irradiating an object to be observed with illumination light to generate image data, and that has a processor that extracts features from the image data and determines the quality of the interference fringe image contained in the image data based on the features.

[0008] The processor is capable of executing reconstruction processing based on the image data, and preferably determines whether or not to execute reconstruction processing based on the result of determining the quality of the interference fringe image.

[0009] The processor preferably extracts the feature quantity by template matching or frequency analysis based on the image data.

[0010] The processor preferably determines the presence or absence of an interference fringe image or a change in the interference fringe image in the image data based on the feature amount.

[0011] It is preferable that the light source has a plurality of light emitting points, and the imaging device performs imaging operations a plurality of times while causing the light emitting points to emit light in sequence, thereby generating a plurality of pieces of image data.

[0012] The processor preferably calculates the feature amount based on the time change of successive interference fringe images included in the plurality of image data.

[0013] The processor preferably calculates a correlation value or a difference value of successive interference fringe images in time series as the feature amount.

[0014] The processor is capable of performing super-resolution processing to generate high-resolution image data based on multiple image data, and reconstruction processing based on the image data generated by the super-resolution processing, and it is preferable to determine whether or not to perform the super-resolution processing and reconstruction processing based on the results of judging the quality of the interference fringe image.

[0015] The object to be observed is preferably a fertilized egg or a floating cell other than a fertilized egg.

[0016] The information processing method disclosed herein is an information processing method that acquires and processes image data from an imaging device that has a light source and an imaging sensor and captures an interference fringe image generated by irradiating an object to be observed with illumination light to generate image data, extracts features from the image data, and determines the quality of the interference fringe image contained in the image data based on the features.

[0017] The program disclosed herein is a program that causes a computer to execute processing by acquiring image data from an imaging device that has a light source and an imaging sensor and captures an interference fringe image generated by irradiating an object to be observed with illumination light to generate image data, and causes the computer to execute processing including extracting feature amounts from the image data and determining the quality of the interference fringe image included in the image data based on the feature amounts. [Effects of the Invention]

[0018] According to the technology of the present disclosure, it is possible to provide an information processing device, an information processing method, and a program that enable improvement in observation efficiency. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a perspective view illustrating an example of an imaging device. [Figure 2] FIG. 2 is a side view of the imaging device on which the culture vessel is placed. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an image sensor. [Figure 4] 10A and 10B are diagrams showing how an interference fringe image is generated by irradiating a fertilized egg with illumination light. [Figure 5] FIG. 1 is a schematic diagram illustrating an example of the configuration of an imaging system. [Figure 6] FIG. 2 is a block diagram showing an example of the internal configuration of an imaging device and an information processing device. [Figure 7] FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device. [Figure 8] FIG. 10 is a diagram illustrating an example of a reconstruction position. [Figure 9] 10A and 10B are diagrams illustrating an example of image data generated by the imaging device when water droplets adhere to the lid. [Figure 10] 10A and 10B are diagrams showing an example of image data generated by an imaging device when air bubbles are mixed into the culture solution. [Figure 11] 10A and 10B are diagrams showing an example of image data generated by an imaging device when there are no water droplets on the lid and no air bubbles in the culture solution. [Figure 12] 10 is a flowchart illustrating an example of the overall operation of the imaging system. [Figure 13] 10 is a diagram showing the configuration of a light emitting surface of a light source included in an imaging device according to a second embodiment. FIG. [Figure 14] FIG. 10 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 15] FIG. 2 is a diagram illustrating a super-resolution process performed by a super-resolution processor. [Figure 16] 10A and 10B are diagrams showing an example in which the pattern of an interference fringe image changes as a fertilized egg rotates. [Figure 17] FIG. 10 is a diagram showing an example in which an interference fringe image moves linearly due to translation of a fertilized egg. [Figure 18] 10 is a flowchart showing an example of the overall operation of the imaging system according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] [First embodiment] 1 shows an example of an imaging device. The imaging device 10 includes a light source 11, an imaging sensor 12, a support 13, a base 14, and a stage 15. The light source 11 is, for example, a laser diode. The imaging device 10 performs so-called lens-free imaging, which captures an image of an object to be observed without using an optical lens.

[0022] The light source 11 may be configured by combining a light emitting diode and a pinhole. The light source 11 emits radial illumination light 16 toward the stage 15. The illumination light 16 is coherent light. The wavelength of the illumination light 16 is 640 nm, 780 nm, or the like.

[0023] Light source 11 is connected to one end of a substantially L-shaped support 13. The other end of support 13 is connected to a base 14. Base 14 is flat and has a stage 15 provided in its approximate center. Stage 15 is provided with a recessed mounting portion 15A on which a culture vessel 20 for culturing a fertilized egg is placed. Support 13 supports light source 11 so that light source 11 faces imaging surface 12A of image sensor 12. A fertilized egg is an example of an "object to be observed" according to the technology of the present disclosure.

[0024] Hereinafter, the direction in which light source 11 and imaging surface 12A face each other will be referred to as the Z direction. The Z direction is also the irradiation direction of illumination light 16. A direction perpendicular to the Z direction will be referred to as the X direction. A direction perpendicular to the Z direction and the X direction will be referred to as the Y direction. Imaging surface 12A is perpendicular to the Z direction and parallel to the X and Y directions.

[0025] The imaging sensor 12 is configured, for example, by a monochrome CMOS (Complementary Metal Oxide Semiconductor) image sensor. The culture vessel 20 is placed on the imaging surface 12A of the imaging sensor 12. The culture vessel 20 is a shallow cylindrical vessel, also called a culture dish. The culture vessel 20 is used together with a lid 25 (see FIG. 2). The culture vessel 20 is transparent and allows the illumination light 16 to pass through. The diameter of the culture vessel 20 is approximately 30 to 60 mm. The thickness of the culture vessel 20 is approximately 10 to 20 mm.

[0026] Fertilized eggs 21 that have been subjected to in vitro fertilization treatment are seeded in culture container 20. In vitro fertilization treatment includes intracytoplasmic sperm injection treatment performed under a microscope and regular in vitro fertilization treatment performed by combining eggs and sperm in a specified container. The method of fertilization of fertilized eggs 21 to be cultured is not important. Fertilized eggs 21 are, for example, human fertilized eggs. Fertilized eggs 21 are approximately spherical and have a diameter of about 100 to 200 μm.

[0027] The fertilized egg 21 is suspended in a culture solution 22 that has been dropped into a culture vessel 20. The culture solution 22 is covered with oil 23 that has been filled into the culture vessel 20. The oil 23 prevents evaporation and changes in pH of the culture solution 22. The fertilized egg 21 in a dividing state is also referred to as an embryo. The fertilized egg 21 in the present disclosure includes an embryo.

[0028] 2 is a side view of the imaging device 10 on which the culture vessel 20 is placed. As shown in FIG. 2, the imaging device 10 captures an image of the fertilized egg 21 with the culture vessel 20 covered with a lid 25.

[0029] The imaging sensor 12 detects illumination light 16 emitted from the light source 11 and transmitted through the culture vessel 20. Specifically, the illumination light 16 enters the culture vessel 20 and is diffracted by the fertilized egg 21, generating an interference fringe image that reflects the shape and internal structure of the fertilized egg 21. The interference fringe image is also called a hologram image. The imaging sensor 12 captures the interference fringe image generated by the fertilized egg 21.

[0030] 3 shows an example of the configuration of the image sensor 12. The image sensor 12 has a plurality of pixels 12B arranged on an imaging surface 12A. The pixels 12B are photoelectric conversion elements that photoelectrically convert incident light and output pixel signals according to the amount of incident light.

[0031] The pixels 12B are arranged at equal pitches along the X and Y directions. The arrangement of the pixels 12B is a so-called square arrangement. The X direction is a direction perpendicular to the Z direction. The Y direction is a direction perpendicular to the X and Z directions. The pixels 12B are arranged at a first arrangement pitch Δx in the X direction and at a second arrangement pitch Δy in the Y direction.

[0032] The imaging sensor 12 captures an image of light incident on the imaging surface 12A, and outputs image data made up of pixel signals output from each of the pixels 12B.

[0033] 4 shows how an interference fringe image is generated by irradiating a fertilized egg 21 with illumination light 16. Part of the illumination light 16 incident on the culture vessel 20 is diffracted by the fertilized egg 21. That is, the illumination light 16 is split into diffracted light 30 that is diffracted by the fertilized egg 21 and transmitted light 31 that is not diffracted by the fertilized egg 21 and passes through the culture vessel 20. The transmitted light 31 is a spherical wave or a plane wave. The diffracted light 30 and transmitted light 31 pass through the bottom surface of the culture vessel 20 and are incident on the imaging surface 12A of the image sensor 12.

[0034] The diffracted light 30 and the transmitted light 31 interfere with each other to generate an interference fringe image 33. The interference fringe image 33 is composed of bright portions 36 and dark portions 38. In FIG. 4, the interference fringe image 33 is illustrated with the bright portions 36 and dark portions 38 each having a circular shape, but the shape of the interference fringe image 33 changes depending on the shape and internal structure of the fertilized egg 21. The imaging sensor 12 captures a light image including the interference fringe image 33 formed on the imaging surface 12A, and outputs image data including the interference fringe image 33.

[0035] Fig. 5 shows an example of the configuration of an imaging system. As shown in Fig. 5, the imaging system 2 includes an image capturing device 10, an incubator 40, and an information processing device 50. The incubator 40 is a multi-room incubator for fertilized eggs, and is also called an embryo culture device. The fertilized eggs 21 are cultured in the incubator 40 for a predetermined period (e.g., 7 days).

[0036] Unlike typical incubators for culturing cells other than fertilized eggs, the incubator 40 does not have one culture chamber, but has multiple culture chambers 41. This is because each culture chamber 41 houses an imaging device 10, allowing the fertilized eggs 21 to be managed individually to prevent them from being mistaken for other people's fertilized eggs 21. The culture chambers 41 are also referred to as culture chambers. Note that the incubator 40 shown in FIG. 5 is provided with two culture chambers 41, but the number of culture chambers 41 is not limited to this and can be changed as appropriate.

[0037] Each culture chamber 41 is provided with an openable / closable lid 42. The incubator 40 is provided with a switch 43 for opening and closing the lid 42 for each culture chamber 41. When the user operates the switch 43, the lid 42 is opened and closed by a drive mechanism (not shown). The lid 42 may also be configured to be opened and closed manually. When the lid 42 is closed, the culture chamber 41 is kept airtight.

[0038] A mixed gas of carbon dioxide (CO2) gas, nitrogen (N2) gas, and outside air is supplied from an external gas cylinder (not shown) to the culture chamber 41 via a HEPA filter (High Efficiency Particulate Air Filter). Heaters (not shown) are provided on the sides and bottom of the culture chamber 41. The culture chamber 41 maintains a constant culture environment by controlling the concentration, temperature, and humidity of the mixed gas to be constant.

[0039] The imaging device 10 is sized to be able to be inserted and removed into the culture chamber 41. As shown in FIG. 5, one imaging device 10 is inserted into one culture chamber 41. That is, the lid 42 can be closed with the imaging device 10 on which the culture container 20 is placed inserted into the culture chamber 41. This allows the fertilized egg 21 to be imaged by the imaging device 10 while being cultured in the culture chamber 41, without having to remove the culture container 20 from the culture chamber 41.

[0040] The information processing device 50 is, for example, a desktop personal computer. A display 51, a keyboard 52, a mouse 53, and the like are connected to the information processing device 50. The keyboard 52 and the mouse 53 constitute an input device 54 for a user to input information. The input device 54 also includes a touch panel and the like.

[0041] The information processing device 50 exchanges data with the imaging devices 10 housed in each of the culture chambers 41 via wireless communication. The imaging devices 10 periodically (for example, every 5 to 15 minutes) capture images. The information processing device 50 periodically receives image data including an interference fringe image 33 (see FIG. 4) from the imaging devices 10, performs reconstruction processing based on the received image data, and displays the reconstructed image generated by the reconstruction processing. The reconstructed image is also called a tomographic image.

[0042] 6 shows an example of the internal configuration of the imaging device 10 and the information processing device 50. As shown in Fig. 6, in addition to the light source 11 and the imaging sensor 12, the imaging device 10 includes a processor 60, a storage device 61, a communication unit 62, a power supply unit 63, and a battery 64, which are interconnected via a bus line 65.

[0043] The processor 60 is, for example, a field programmable gate array (FPGA) and controls the operation of each unit within the imaging device 10. The storage device 61 is, for example, a random access memory (RAM) or a flash memory. The storage device 61 stores image data generated by the imaging device 10 and various other data.

[0044] The communication unit 62 performs wireless communication with the information processing device 50. The processor 60 transmits image data to the information processing device 50 via the communication unit 62.

[0045] The battery 64 is a secondary battery such as a lithium polymer battery. The power supply unit 63 includes a power supply circuit and a charge control circuit. The power supply unit 63 supplies power supplied from the battery 64 to the processor 60 and the like. The power supply unit 63 also controls charging of the battery 64 using power supplied from an external source. The power supply unit 63 may be configured to be able to charge the battery 64 wirelessly.

[0046] The information processing device 50 includes a processor 55, a storage device 56, and a communication unit 57, which are interconnected via a bus line 58. The bus line 58 is also connected to the display 51 and input device 54 described above.

[0047] The processor 55 is configured by, for example, a CPU (Central Processing Unit), and realizes various functions by reading out an operating program 56A and various data stored in the storage device 56 and executing the processes.

[0048] The storage device 56 includes, for example, RAM, ROM (Read Only Memory), or a storage device. The RAM is, for example, a volatile memory used as a work area, etc. The ROM is, for example, a non-volatile memory such as a flash memory that stores the operating program 56A and various data. The storage device is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage stores an OS (Operating System), application programs, image data, various data, etc.

[0049] The communication unit 57 performs wireless communication with the communication unit 62 of the imaging device 10. The processor 55 receives image data transmitted from the imaging device 10 via the communication unit 57. The processor 55 also transmits a control signal for controlling imaging to the imaging device 10 via the communication unit 57.

[0050] The display 51 displays various screens. The information processing device 50 receives input of operation instructions from the input device 54 via the various screens.

[0051] 7 shows an example of the functional configuration of the information processing device 50. The functions of the information processing device 50 are realized by the processor 55 executing processing based on the operating program 56A. As shown in FIG. 7, the processor 55 includes an imaging control unit 70, an image data acquisition unit 71, a reconstruction processing unit 72, a display control unit 73, a feature extraction unit 74, and a determination unit 75.

[0052] The imaging control unit 70 controls the operation of the imaging device 10. Specifically, the imaging control unit 70 controls the operation of generating illumination light 16 by the light source 11 and the imaging operation of the imaging sensor 12 by transmitting a control signal to the imaging device 10. Hereinafter, the operation of generating illumination light 16 by the light source 11 and the imaging operation of the imaging sensor 12 will be collectively referred to as the imaging operation of the imaging device 10. The imaging control unit 70 causes the imaging device 10 to start the imaging operation based on an operation signal input from the input device 54.

[0053] The image data acquisition unit 71 acquires image data generated and transmitted from the imaging device 10 after the imaging device 10 captures an image of the fertilized egg 21 in the culture container 20. The image data acquisition unit 71 supplies the acquired image data to the reconstruction processing unit 72.

[0054] The reconstruction processing unit 72 generates a reconstructed image by performing calculations based on the image data. For example, as shown in Fig. 8, the reconstruction processing unit 72 changes the reconstruction position P in the Z direction by a fixed amount, and generates a reconstructed image each time the reconstruction position P is changed. The reconstruction position P is a position (so-called depth position) represented by a distance d from the imaging surface 12A of the image sensor 12 toward the light source 11. Hereinafter, the reconstruction position P will also be referred to as the focal position.

[0055] The reconstruction processing unit 72 performs reconstruction processing based on the Fresnel transformation formulas expressed by the following formulas (1) to (3), for example.

[0056]

number

[0057]

number

[0058]

number

[0059] Here, I(x, y) represents image data. x represents the coordinate of pixel 12B of image sensor 12 (see FIG. 3) in the X direction. y represents the coordinate of pixel 12B in the Y direction. Δx is the first array pitch mentioned above, and Δy is the second array pitch mentioned above (see FIG. 3). λ is the wavelength of illumination light 16.

[0060] As shown in equation (1), Γ(m,n) is a complex amplitude image obtained by Fresnel transforming the interference fringe image contained in the image data. Here, m = 1, 2, 3, . . . Nx-1 and n = 1, 2, 3, . . . Ny-1. Nx represents the number of pixels in the X direction of the image data. Ny represents the number of pixels in the Y direction of the image data.

[0061] As shown in equation (2), A0(m,n) is an intensity distribution image representing the intensity components of the complex amplitude image Γ(m,n). As shown in equation (3), φ0(m,n) is a phase distribution image representing the phase components of the complex amplitude image Γ(m,n).

[0062] The reconstruction processing unit 72 obtains a complex amplitude image Γ(m,n) based on equation (1), and obtains an intensity distribution image A0(m,n) or a phase distribution image φ0(m,n) by applying the obtained complex amplitude image Γ(m,n) to equation (2) or equation (3). The reconstruction processing unit 72 obtains either the intensity distribution image A0(m,n) or the phase distribution image φ0(m,n) and outputs it as a reconstructed image.

[0063] In this embodiment, the reconstruction processing unit 72 outputs the phase distribution image φ0(m,n) as the reconstructed image. The phase distribution image φ0(m,n) is an image that represents the refractive index distribution of the object being observed. Since the fertilized egg 21, which is the object being observed in this embodiment, is semitransparent, most of the illumination light 16 is not absorbed by the fertilized egg 21 but is transmitted or diffracted, and therefore almost no image appears in the intensity distribution. For this reason, in this embodiment, it is preferable to use the phase distribution image φ0(m,n) as the reconstructed image.

[0064] The reconstruction processing unit 72 is not limited to a method using the Fresnel transform formula, and may perform reconstruction processing using a Fourier iterative phase retrieval method or the like.

[0065] The display control unit 73 causes the reconstructed image generated by the reconstruction processing unit 72 to be displayed on the display 51. The reconstructed image at one focal position may be displayed on the display 51, or the reconstructed images at multiple focal positions may be displayed. Furthermore, the focal position of the reconstructed image displayed on the display 51 may be set or selected by the user operating the input device 54.

[0066] Because the fertilized egg 21 has a thickness of approximately 100 to 200 μm and is suspended in the culture solution 22, it is difficult to adjust the focal position on the pronucleus and other components inside the fertilized egg 21 using conventional microscope observation. For this reason, for example, in the conventional technology described in Japanese Patent Application Laid-Open No. 2018-093795, multiple images are captured at different focal positions. In contrast, the lens-free imaging of the present disclosure makes it possible to generate a reconstructed image at any focal position based on image data obtained in a single capture.

[0067] The feature extraction unit 74 extracts image features from the image data acquired by the image data acquisition unit 71. For example, the feature extraction unit 74 searches for the interference fringe image 33 from the image data by performing template matching on the image data using the interference fringe image 33 shown in Fig. 4 as a template. The feature extraction unit 74 outputs a correlation value indicating the degree of matching between the image included in the image data and the interference fringe image 33 to the determination unit 75 as a feature.

[0068] The determination unit 75 determines the quality of the interference fringe image included in the image data acquired by the image data acquisition unit 71 based on the feature amount input from the feature amount extraction unit 74. Here, "quality" corresponds to the degree of similarity with the interference fringe image 33 obtained by imaging the fertilized egg 21. The quality of the interference fringes to be determined is higher the higher the similarity with the interference fringe image 33 of the fertilized egg 21. For example, when the feature amount represents a correlation value of template matching, the determination unit 75 determines the quality of the interference fringe image based on whether the correlation value is equal to or greater than a reference value.

[0069] The determination unit 75 outputs a determination result R of the quality of the interference fringe image to the reconstruction processing unit 72 and the display control unit 73. The determination result R includes information indicating whether or not to continue the reconstruction processing by the reconstruction processing unit 72, depending on the quality of the interference fringe image. For example, the determination result R includes a permission signal R1 that permits the execution of the reconstruction processing when the quality of the interference fringe image is equal to or higher than a certain level (i.e., the correlation value is equal to or higher than a reference value), or a non-permission signal R2 that does not permit the execution of the reconstruction processing when the quality of the interference fringe image is lower than a certain level (i.e., the correlation value is lower than a reference value).

[0070] The reconstruction processing unit 72 determines whether or not to execute the reconstruction process based on the determination result R input from the determination unit 75. The reconstruction processing unit 72 executes the reconstruction process if the determination result R includes a permission signal R1. On the other hand, the reconstruction processing unit 72 does not execute the reconstruction process if the determination result R includes a non-permission signal R2.

[0071] The display control unit 73 causes the display 51 to display a message based on the determination result R input from the determination unit 75. For example, when the determination result R includes a disallowance signal R2, the display control unit 73 causes the display 51 to display a message indicating that the reconstruction process will not be performed because the quality of the interference fringe image is low. In this case, it is preferable that the display control unit 73 causes the display 51 to display a message urging the user to perform re-imaging. In addition, in this case, since there is a possibility that the quality of the interference fringe image has deteriorated due to the adhesion or inclusion of foreign matter in the incubation vessel 20 or the lid 25, it is preferable that the display control unit 73 causes the display 51 to display a message urging the user to check the condition of the object to be observed before performing re-imaging.

[0072] 9 to 11 show examples of a determination result R obtained by determining the quality of an interference fringe image. FIG. 9 shows an example of image data generated by the imaging device 10 when a water droplet 80 is attached to the lid 25. As shown in FIG. 9, when a water droplet 80 is attached to the lid 25, the illumination light 16 is diffused by being incident on the water droplet 80, resulting in diffused light. In this case, the interference fringe image 33 does not appear in the image data D, and diffused light that is diffused throughout is captured. In this case, the correlation value calculated by template matching as the feature extracted by the feature extractor 74 is less than the reference value. As a result, the determiner 75 determines that the quality of the interference fringe image is less than a certain level, and outputs a determination result R including a disallowance signal R2 to the reconstruction processor 72 and the display controller 73.

[0073] FIG. 10 shows an example of image data generated by the imaging device 10 when an air bubble 81 is mixed into the culture solution 22. As shown in FIG. 10, when an air bubble 81 is present in the culture solution 22, the difference in refractive index between the culture solution 22 and the air bubble 81 (i.e., air) causes the influence of light refraction. In this case, the interference fringe image 33 does not appear in the image data D, and only the shadow 81A of the air bubble 81 is captured. In this case, the correlation value calculated by template matching as the feature extracted by the feature extractor 74 is less than the reference value. As a result, the determiner 75 determines that the quality of the interference fringe image is below a certain level, and outputs a determination result R including a disallowance signal R2 to the reconstruction processor 72 and the display controller 73.

[0074] 11 shows an example of image data generated by the imaging device 10 when there are no water droplets on the lid 25, no air bubbles in the culture solution 22, etc. As shown in FIG. 11, when the fertilized egg 21 is imaged satisfactorily, the image data D includes an interference fringe image 33 based on the fertilized egg 21. In this case, the correlation value calculated by template matching as the feature extracted by the feature extractor 74 is equal to or greater than the reference value. As a result, the determiner 75 determines that the quality of the interference fringe image is equal to or greater than a certain level, and outputs a determination result R including an enable signal R1 to the reconstruction processor 72 and the display controller 73.

[0075] In other words, in this embodiment, the determination unit 75 determines whether or not there is an interference fringe image 33 in the image data, and outputs a determination result R according to the presence or absence of the interference fringe image 33. That is, in this embodiment, whether or not to execute the reconstruction process is determined based on the presence or absence of the interference fringe image 33.

[0076] Next, an example of the overall operation of the imaging system 2 will be described with reference to the flowchart shown in Fig. 12. First, the user places the culture vessel 20 on the stage 15 of the imaging device 10, and then inserts the imaging device 10 into the culture chamber 41 of the incubator 40. Note that it is sufficient to insert at least one imaging device 10 into each of the multiple culture chambers 41.

[0077] Next, the user closes the lid 42 of the culture chamber 41 and starts culture in the incubator 40. When the incubator 40 starts culture, the imaging device 10 captures an image of the fertilized egg 21 in the culture container 20 under the control of the information processing device 50 (step S10). The imaging device 10 transmits image data generated by performing the imaging operation to the information processing device 50.

[0078] The information processing device 50 acquires image data transmitted from the imaging device 10 via the image data acquisition unit 71 (step S11). The feature extraction unit 74 of the information processing device 50 extracts feature amounts from the image data acquired by the image data acquisition unit 71 (step S12). For example, the feature extraction unit 74 extracts a correlation value representing the degree of matching with the interference fringe image 33 as a feature amount by template matching.

[0079] Next, the determination unit 75 determines the quality of the interference fringe image contained in the image data based on the features extracted by the feature extraction unit 74, and outputs the determination result R to the reconstruction processing unit 72 and the display control unit 73 (step S13).

[0080] If the quality of the interference fringe image is equal to or higher than a certain level (step S14: YES), an enable signal R1 is output from the determination unit 75 to the reconstruction processing unit 72 and the display control unit 73 as the determination result R. In this case, the reconstruction processing unit 72 generates a reconstructed image by performing reconstruction processing based on the image data acquired by the image data acquisition unit 71 (step S15). The display control unit 73 displays the reconstructed image generated by the reconstruction processing unit 72 on the display 51 (step S16).

[0081] On the other hand, if the quality of the interference fringe image is below a certain level (step S14: NO), a disallowance signal R2 is output from the determination unit 75 to the reconstruction processing unit 72 and the display control unit 73 as the determination result R. In this case, the reconstruction processing unit 72 does not execute the reconstruction process. The display control unit 73 displays a message on the display 51 indicating that the reconstruction process will not be executed due to the low quality of the interference fringe image (step S17).

[0082] As described above, the imaging system 2 according to the embodiment determines the quality of the interference fringe image included in the image data obtained by the imaging device 10 performing imaging, and does not execute reconstruction processing if the quality is below a certain level. Therefore, if the quality of the interference fringe image is low, the user does not need to wait for the reconstruction processing to finish, and can quickly execute re-imaging after checking whether any foreign matter has adhered to or been mixed into the incubation container 20 or the lid 25. This improves the efficiency of observing the object being observed.

[0083] In the above embodiment, the feature extraction unit 74 extracts a correlation value representing the degree of matching with the interference fringe image 33 as a feature by template matching, but the feature extraction unit 74 may extract the frequency characteristics of the image data as a feature. In this case, for example, the feature extraction unit 74 extracts the frequency characteristics by performing frequency analysis on the image data using a Fourier transform or the like.

[0084] As shown in Fig. 11, a plurality of specific frequencies are extracted in the high frequency region from image data D obtained by capturing an image of fertilized egg 21 due to the striped pattern of interference fringe image 33. In contrast to this, as shown in Fig. 9, when water droplets 80 adhere to lid 25, certain frequency characteristics due to diffused light are extracted from image data D. Furthermore, as shown in Fig. 10, when air bubbles 81 are mixed into culture solution 22, frequency characteristics in the low frequency region due to shadows 81A of the air bubbles 81 are extracted from image data D.

[0085] In this case, the determination unit 75 determines the quality of the interference fringe image based on the frequency characteristics as a feature amount. The determination unit 75 determines that the closer the frequency characteristics extracted from the image data D to be determined are to the frequency characteristics of the interference fringe image 33 of the fertilized egg 21, the higher the quality.

[0086] [Second embodiment] Next, a second embodiment will be described. In the second embodiment, the quality of an interference fringe image is determined based on a plurality of image data obtained by capturing images multiple times.

[0087] In the second embodiment, the light source 11 may be a laser light source in which a plurality of light-emitting points (for example, 36 light-emitting points) are arranged in a two-dimensional array. A vertical cavity surface-emitting laser (VCSL) can be used as this laser light source. By synthesizing a plurality of image data obtained by the imaging sensor 12 performing an imaging operation while the plurality of light-emitting points are sequentially emitting light, image data including a high-resolution interference fringe image (so-called super-resolution interference fringe image) can be obtained. By reconstructing this image data, a high-quality reconstructed image can be obtained.

[0088] FIG. 13 shows the configuration of a light emitting surface 11A of a light source 11 included in an imaging device 10 according to the second embodiment. The light emitting surface 11A is disposed in a position facing the imaging sensor 12. A plurality of light emitting points 11B are arranged in a two-dimensional array on the light emitting surface 11A. The arrangement pitch of the light emitting points 11B is approximately 10 μm to 100 μm. Each of the light emitting points 11B is selected in turn to emit illumination light 16. The light emitting time interval between the plurality of light emitting points 11B is several milliseconds.

[0089] The arrangement pitch of the light-emitting points 11B only needs to be different from the arrangement pitch of the pixels 12B (first arrangement pitch Δx and second arrangement pitch Δy), and does not necessarily need to be smaller than the arrangement pitch of the pixels 12B. For example, even if a light-emitting point 11B is located directly above an adjacent pixel 12B, the arrangement pitch of the light-emitting points 11B does not need to match the arrangement pitch of the pixels 12B. In this case, the illumination light 16 is illuminated at different positions on the pixels 12B. Therefore, when combining multiple image data, different pixels 12B that are located directly below each light-emitting point 11B and illuminated by the illumination light 16 are considered to be the same pixel, and alignment with an accuracy of one pixel or less can be performed, thereby generating image data including a super-resolution interference fringe image.

[0090] 13, the light-emitting points 11B are arranged in a 6 × 6 square array, and 36 light-emitting points 11B are provided on the light-emitting surface 11A, but the number and arrangement pattern of the light-emitting points 11B are not limited to the number and arrangement pattern shown in Fig. 13. The more light-emitting points 11B there are, the higher the resolution of the interference fringe image can be, but the longer the calculation time for the synthesis process and reconstruction process. For this reason, it is preferable to optimize the number of light-emitting points 11B according to the required image quality and calculation time.

[0091] The image sensor 12 performs an image capturing operation and generates image data each time each light emitting point 11B emits light. The image capturing device 10 transmits a plurality of image data obtained by the image sensor 12 capturing images a plurality of times to the information processing device 50.

[0092] Fig. 14 shows an example of the functional configuration of the information processing device 50 according to the second embodiment. As shown in Fig. 14, in the second embodiment, the processor 55 includes an imaging control unit 70, an image data acquisition unit 71, a reconstruction processing unit 72, a display control unit 73, a feature extraction unit 74, and a determination unit 75, as well as a super-resolution processing unit 76.

[0093] Fig. 15 schematically shows the super-resolution processing by the super-resolution processing unit 76. As shown in Fig. 15, a plurality of light-emitting points 11B provided in the light source 11 emit light one by one in sequence under control of the imaging control unit 70. In Fig. 15, t represents time. The light-emitting points 11B that emit light at each time t in the time series t=t1, t2, t3, ... are different from each other.

[0094] The interference fringe images 33 included in the image data D obtained by the imaging sensor 12 performing imaging at each time t in the time series t=t1, t2, t3, and so on are shifted in position within the image data D because the light-emitting points 11B are different. The super-resolution processor 76 aligns and combines the multiple image data D to generate image data DS including a super-resolution interference fringe image 33S. The super-resolution processor 76 outputs the generated image data DS to the reconstruction processor 72.

[0095] 16 and 17 illustrate the processing of the feature extraction unit 74 and the determination unit 75 according to the second embodiment. In this embodiment, the feature extraction unit 74 calculates, as a feature, a correlation value between successive interference fringe images in time series for a plurality of image data D. The determination unit 75 determines whether the correlation value is equal to or greater than a reference value. If the correlation value is equal to or greater than the reference value, the determination unit 75 determines that the quality of the interference fringe image is above a certain level, and outputs a determination result R including an acceptance signal R1. If the correlation value is less than the reference value, the determination unit 75 determines that the quality of the interference fringe image is below a certain level, and outputs a determination result R including a rejection signal R2.

[0096] The feature extraction unit 74 may calculate, as a feature, not only the correlation value of chronologically consecutive interference fringe images but also the difference value of chronologically consecutive interference fringe images. Even when the fertilized egg 21 moves in parallel, the difference value can be calculated by aligning the chronologically consecutive interference fringe images and then taking the difference. In other words, the feature extraction unit 74 may calculate a feature based on the temporal change of the chronologically consecutive interference fringe images.

[0097] 16 and 17 show examples of calculating a correlation value between image data D obtained at time t1 and image data D obtained at time t2.

[0098] FIG. 16 shows an example in which the pattern of the interference fringe image 33 in the image data D changes due to the rotation of the fertilized egg 21 between time t1 and time t2. The fertilized egg 21 is prone to rotation because it is suspended in the culture solution 22. When the fertilized egg 21 rotates, the position of the pronucleus and other components inside the fertilized egg 21 changes, causing a change in the pattern of the interference fringe image 33. When the pattern of the interference fringe image 33 changes due to the rotation of the fertilized egg 21 in this way, the correlation value calculated by the feature extraction unit 74 becomes less than the reference value. In this case, the determination unit 75 determines that the quality of the interference fringe image is below a certain level and outputs a determination result R including a disallowance signal R2.

[0099] If super-resolution processing were performed using multiple image data D with such changed interference fringe images 33, the image data DS generated by the super-resolution processing would have a lower image quality. For this reason, in this embodiment, the execution of super-resolution processing is prohibited when it is expected that the image quality of the image data DS will be lowered based on the correlation value as described above.

[0100] 17 shows an example in which the interference fringe image 33 in the image data D moves linearly due to translation of the embryo 21 between time t1 and time t2. When the embryo 21 moves parallel, no change occurs in the pattern of the interference fringe image 33. In this way, when the interference fringe image 33 moves linearly due to translation of the embryo 21, the correlation value calculated by the feature extraction unit 74 is equal to or greater than the reference value. In this case, the determination unit 75 determines that the quality of the interference fringe image is above a certain level, and outputs a determination result R including an enable signal R1.

[0101] Even if the interference fringe image 33 moves linearly, the super-resolution processing unit 76 can align the interference fringe image 33, so the image quality of the image data DS generated by the super-resolution processing unit 76 does not decrease.

[0102] The pattern of the interference fringe image 33 may be generated not only by the rotation of the fertilized egg 21 but also by cell division of the fertilized egg 21. Also, immediately after inserting the imaging device 10 into the incubator 40, a large temperature change occurs, which may cause a large movement of the fertilized egg 21 and change the pattern of the interference fringe image 33. Thus, in this embodiment, whether or not to perform the super-resolution processing and the reconstruction processing is determined based on the change in the pattern of the interference fringe image 33 over time.

[0103] Next, an example of the overall operation of the imaging system 2 according to the second embodiment will be described with reference to the flowchart shown in Fig. 18. In this embodiment, after starting the culture as in the first embodiment, multiple images are captured while changing the light emitting point 11B of the light source 11 (step S20). The imaging device 10 transmits multiple image data generated by performing multiple imaging operations to the information processing device 50.

[0104] The information processing device 50 acquires the plurality of image data transmitted from the imaging device 10 by the image data acquisition unit 71 (step S21). The feature extraction unit 74 of the information processing device 50 extracts feature values ​​from the plurality of image data acquired by the image data acquisition unit 71 (step S22). In this embodiment, for each of the plurality of image data, correlation values ​​of chronologically consecutive interference fringe images are calculated as feature values.

[0105] Next, the determination unit 75 determines the quality of the interference fringe image included in the image data based on the features extracted by the feature extraction unit 74, and outputs a determination result R to the super-resolution processing unit 76, the reconstruction processing unit 72, and the display control unit 73 (step S23). In this embodiment, if at least one of the multiple correlation values ​​calculated by the feature extraction unit 74 is less than a reference value, the determination unit 75 determines that the quality of the interference fringe image is less than a certain level, and outputs a determination result R including a disallowance signal R2. On the other hand, if all of the multiple correlation values ​​calculated by the feature extraction unit 74 are equal to or greater than the reference values, the determination unit 75 determines that the quality of the interference fringe image is equal to or greater than a certain level, and outputs a determination result R including an allowance signal R1.

[0106] If the quality of the interference fringe image is above a certain level (step S24: YES), the super-resolution processor 76 performs super-resolution processing based on the multiple image data acquired by the image data acquisition unit 71 to generate image data including a super-resolution interference fringe image (step S25). The reconstruction processor 72 performs reconstruction processing based on the image data generated by the super-resolution processor 76 to generate a reconstructed image (step S26). The display controller 73 causes the display 51 to display the reconstructed image generated by the reconstruction processor 72 (step S27).

[0107] On the other hand, if the quality of the interference fringe image is below a certain level (step S24: NO), the super-resolution processor 76 does not perform super-resolution processing, and the reconstruction processor 72 does not perform reconstruction processing. In this case, the display controller 73 causes the display 51 to display a message indicating that the reconstruction processing will not be performed due to the low quality of the interference fringe image (step S28).

[0108] As described above, the imaging system 2 according to this embodiment determines the quality of the interference fringe image based on the temporal change of the interference fringe image contained in the multiple image data obtained by the imaging device 10, and does not execute the super-resolution processing and reconstruction processing if the quality is below a certain level. Therefore, if the quality of the interference fringe image is low, the user does not need to wait for the super-resolution processing and reconstruction processing to finish, and can quickly re-image the image after checking the state of the fertilized egg 21. This improves the efficiency of observing the object being observed.

[0109] In the second embodiment, the feature extraction unit 74 and the determination unit 75 perform processing after the imaging device 10 generates multiple pieces of image data through multiple imaging operations, but they may also perform processing each time the imaging device 10 performs one imaging operation and generates one piece of image data. That is, while the imaging device 10 is performing multiple images, it may be determined whether or not to continue the imaging operation based on the determination result of the quality of the interference fringe image. In this case, the imaging operation is stopped when it is determined that the quality of the interference fringe image is below a certain level.

[0110] Furthermore, in the first and second embodiments, when the imaging system 2 performs imaging and reconstruction processing every fixed time (for example, every 10 minutes) (i.e., when performing time-lapse imaging), it may be determined whether to continue the imaging and reconstruction processing based on the determination result of the quality of the interference fringe image. In this case, when it is determined that the quality of the interference fringe image is below a certain level, the imaging and reconstruction processing are stopped.

[0111] In the first and second embodiments, the imaging device 10 and the information processing device 50 are separate devices, but the imaging device 10 and the information processing device 50 may be integrated into one device. Furthermore, the imaging device 10, the information processing device 50, and the incubator 40 may be integrated into one device.

[0112] Furthermore, in the first and second embodiments, the object of observation is a fertilized egg, but the object of observation may be a floating cell other than a fertilized egg. A floating cell is a cell that floats in a culture medium. In addition to a fertilized egg, floating cells include CHO (Chinese Hamster Ovary) cells used for antibody production.

[0113] The imaging system 2 according to the above embodiment relates to a technique called lens-free imaging, in which the imaging device 10 does not include an optical lens. The technique of the present disclosure is applicable to digital holography in general (for example, when a reference light is used).

[0114] Various modifications are possible to the hardware configuration of the computer that constitutes the information processing device 50. For example, the information processing device 50 can be configured with multiple computers that are separated as hardware in order to improve processing capacity and reliability.

[0115] In this way, the hardware configuration of the computer of the information processing device 50 can be changed as appropriate depending on the required performance such as processing power, safety, reliability, etc. Furthermore, not only the hardware but also application programs such as the operating program 56A can be duplicated or stored in a distributed manner across multiple storage devices in order to ensure safety and reliability.

[0116] In each of the above embodiments, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the imaging control unit 70, image data acquisition unit 71, reconstruction processing unit 72, display control unit 73, feature extraction unit 74, determination unit 75, and super-resolution processing unit 76. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (operation program 56A) and functions as various processing units, as well as dedicated electrical circuits, such as programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as FPGAs, and application specific integrated circuits (ASICs), which are processors with a circuit configuration designed specifically for performing specific processes.

[0117] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0118] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0119] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0120] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction.

[0121] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An information processing device that includes a light source having a plurality of light-emitting points and an imaging sensor, and that captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, and acquires and processes the image data from the imaging device, a processor; The processor: extracting, as a feature, a similarity between a plurality of image data pieces generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image included in the image data based on the feature amount; Information processing device.

2. The similarity is a correlation value of time-series successive interference fringe images for the plurality of image data. The information processing device according to claim 1 .

3. The processor, It is possible to perform a super-resolution process for generating high-resolution image data based on the plurality of image data, and a reconstruction process based on the image data generated by the super-resolution process, determining whether or not to execute the super-resolution processing and the reconstruction processing based on the result of the determination of the quality of the interference fringe image; 3. The information processing device according to claim 1.

4. The observation object is a fertilized egg or a floating cell other than a fertilized egg. The information processing device according to any one of claims 1 to 3.

5. An information processing device that includes a light source having a plurality of light-emitting points and an imaging sensor, and that captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, and acquires and processes the image data from the imaging device, a processor; The processor: calculating a feature amount based on a time change of a time-series successive interference fringe image included in a plurality of image data generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image based on the feature amount; Information processing device.

6. the processor calculates a correlation value or a difference value of time-series successive interference fringe images as a feature amount; The information processing device according to claim 5 .

7. The processor: It is possible to perform a super-resolution process for generating high-resolution image data based on the plurality of image data, and a reconstruction process based on the image data generated by the super-resolution process, determining whether or not to execute the super-resolution processing and the reconstruction processing based on the result of the determination of the quality of the interference fringe image; 7. The information processing device according to claim 5 or 6.

8. The object to be observed is a fertilized egg or a floating cell other than a fertilized egg. The information processing device according to any one of claims 1 to 7.

9. An information processing method for acquiring and processing image data from an imaging device that is equipped with a light source having a plurality of light-emitting points and an imaging sensor, and that captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, comprising: extracting, as a feature, a similarity between a plurality of image data pieces generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image included in the image data based on the feature amount; Information processing methods.

10. A program for causing a computer to execute a process of acquiring image data from an imaging device that has a light source having a plurality of light-emitting points and an imaging sensor, and captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, the program comprising: extracting, as a feature, a similarity between a plurality of pieces of image data generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image included in the image data based on the feature amount; A program that causes a computer to execute a process including the above.

11. An information processing method for acquiring and processing image data from an imaging device that is equipped with a light source having a plurality of light-emitting points and an imaging sensor, and that captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, comprising: calculating a feature amount based on a time change of a time-series successive interference fringe image included in a plurality of image data generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image based on the feature amount; Information processing methods.

12. A program for acquiring image data from an imaging device that has a light source having a plurality of light-emitting points and an imaging sensor, and that captures an interference fringe image generated by irradiating an observation object with illumination light to generate image data, and causing a computer to execute processing, calculating a feature amount based on a time change of a time-series successive interference fringe image included in a plurality of image data generated by the imaging device performing imaging operations a plurality of times while sequentially emitting light from the plurality of light-emitting points; determining the quality of the interference fringe image based on the feature amount; A program that causes a computer to execute a process including the above.

Images