Imaging method and imaging system

Abstract

This imaging method includes: irradiating an eye of a subject in a dark room with first light; and generating first image data that represents the fundus of the subject and includes information pertaining to four or more wavelength bands. The first image data is generated by snapshot-imaging the eye of the subject by using a camera before pupil contraction of the eye caused by the irradiation with the first light is completed.

Description

Imaging method and imaging system

[0001] The present disclosure relates to an imaging method for imaging a fundus and an imaging system for performing the imaging method.

[0002] Fundus cameras are used to diagnose various diseases that occur in various biological elements, such as the retina, optic nerve, and blood vessels. Each element is clearly imaged in a different wavelength band. For example, Patent Document 1 describes that illumination light with a wavelength of 550 nm to 600 nm is effective for imaging arteries and veins, illumination light with a wavelength of 620 nm to 690 nm is effective for imaging the optic nerve head, and illumination light with a wavelength of 660 nm to 720 nm is effective for imaging the choroid. It also describes that arteries and veins can be distinguished by comparing an image with a wavelength of 550 nm with an image with a wavelength of 580 nm.

[0003] Patent Document 2 describes that illumination light with a wavelength of 710 nm to 750 nm is effective for imaging subretinal choroidal blood vessels, that illumination light with a wavelength of 790 nm to 830 nm is effective for imaging both epiretinal blood vessels and subretinal choroidal blood vessels, and that illumination light with a wavelength of 900 nm to 1000 nm is effective for imaging epiretinal blood vessels.

[0004] In addition to these narrow wavelength band imaging, imaging using illumination light that removes components with wavelengths above approximately 575 nm is called red-free imaging, and is effective in diagnosing optic nerve fiber-related diseases such as glaucoma. For this reason, many fundus cameras are equipped with filters for this purpose.

[0005] Acquiring images of the fundus in multiple wavelength bands can provide information that is useful for diagnosing multiple diseases. An imaging method that acquires images in four or more wavelength bands is called hyperspectral imaging, and a camera that performs this method is called a hyperspectral camera.

[0006] As a method for hyperspectral imaging, Patent Document 1 discloses a method in which the fundus is irradiated with a broadband light source, a liquid crystal tunable filter capable of changing the transmission characteristics is arranged in the imaging section, and multiple images are taken while the transmission characteristics are sequentially changed. Patent Document 3 discloses a method in which a tunable wavelength bandpass filter capable of changing the transmission characteristics is arranged in the light source, and multiple images are taken while the wavelength range of the illumination light is sequentially changed.

[0007] Patent No. 4854389 Publication Patent No. 5170625 Publication Special Publication No. 2023-534401

[0008] The present disclosure provides an imaging method for hyperspectral fundus imaging that acquires images in multiple wavelength bands without shifting the imaging position.

[0009] An imaging method according to one aspect of the present disclosure includes irradiating a subject's eye in a darkroom with a first light and generating first image data representing the fundus of the subject, the first image data including information of four or more wavelength bands, wherein the first image data is generated by having a camera take a snapshot of the subject's eye before pupil constriction of the eye caused by the irradiation of the first light ends.

[0010] A comprehensive or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. When an apparatus consists of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices. The multiple devices included in a "system" may include devices installed in remote locations away from other devices and connected via a communication network.

[0011] According to the present disclosure, hyperspectral fundus imaging can acquire images in multiple wavelength bands without shifting the imaging position, thereby enabling easy and accurate comparison of images across multiple wavelength bands.

[0012] FIG. 1 schematically illustrates the configuration of a non-mydriatic fundus camera according to a first embodiment of the present disclosure. FIG. 2 schematically illustrates an example of the arrangement of filter regions. FIG. 3 schematically illustrates example transmission characteristics of filter regions. FIG. 4 schematically illustrates the configuration of a non-mydriatic fundus camera according to a second embodiment of the present disclosure. FIG. 5 schematically illustrates an example of the spatial distribution of transmittance for each wavelength band of a coded mask. FIG. 6A illustrates an example of a first transmission spectrum of a first filter region. FIG. 6B illustrates an example of a second transmission spectrum of a second filter region. FIG. 7 illustrates a flowchart for performing reflective hyperspectral imaging. FIG. 8 illustrates a timing chart for performing reflective hyperspectral imaging. FIG. 9 illustrates a flowchart for performing both reflective hyperspectral imaging and autofluorescence imaging. FIG. 10 illustrates the timing for performing both reflective hyperspectral imaging and autofluorescence imaging. FIG. 11 schematically illustrates an example of spectra related to identifying fluorescence spectra. FIG. 12 illustrates a timing chart for performing reflective hyperspectral imaging. FIG. 13 shows the timing for both reflectance hyperspectral imaging and autofluorescence imaging.

[0013] In the present disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which allows reconfiguration of the connections within the LSI or the setup of circuit sections within the LSI, can also be used for the same purpose.

[0014] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices, such as interfaces.

[0015] In this disclosure, "light" refers to electromagnetic waves including not only visible light (wavelength of about 400 nm to about 700 nm), but also ultraviolet light (wavelength of about 10 nm to about 400 nm) and infrared light (wavelength of about 700 nm to about 1 mm). In this specification, ultraviolet light may be referred to as "ultraviolet light," and infrared light may be referred to as "infrared light."

[0016] Exemplary embodiments of the present disclosure will be described below. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts will be described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0017] (Findings that form the basis of the present disclosure) In the hyperspectral imaging methods of Patent Documents 1 and 3, images of multiple wavelength bands are acquired sequentially, so the images of the multiple wavelength bands are not synchronized. The subject of the fundus camera is a human body, which may be constantly moving.

[0018] It is not easy to directly compare images that are not simultaneous. In particular, to distinguish elements by comparing the intensity of images in two wavelength bands, as in the method of Patent Document 1, precise alignment of the imaging positions is required. Patent Document 1 discloses image processing that corrects the imaging position, but the load of this image processing increases with the number of images captured. In other words, the greater the number of wavelength bands in hyperspectral imaging, the heavier the image processing load. Due to the curved shape of the eyeball, imaging results from different positions away from the optical axis of the optical system are affected by various aberrations such as field distortion, differences in focus, and vignetting in different states. Therefore, it is not easy to perform a strict comparison using simple corrections such as translation and / or rotation of the image.

[0019] The methods of Patent Documents 1 and 3 also have limitations on non-mydriatic fundus imaging, which does not require the use of mydriatic drugs and places less strain on the subject. Non-mydriatic fundus imaging is an imaging method that completes imaging within approximately 0.2 seconds from the irradiation of illumination light containing visible light components that induce the miosis reflex until the pupil contracts and imaging becomes difficult.

[0020] The first limitation of applying conventional methods to non-mydriatic fundus imaging is the light source. The hyperspectral imaging methods described in Patent Documents 1 and 3 require that the light source spectrum be constant while imaging each wavelength band. If the light source spectrum does not match between wavelength bands, the imaging results will contain variations in the light source spectrum, making them indistinguishable from the differences in the spectrum of the subject that is the intended subject. The method described in Patent Document 3 addresses this issue by using dedicated LED (light-emitting diode) illumination. Because LEDs have a narrow wavelength band, an optical system is used that mixes the light from multiple LEDs to obtain illumination components from the visible to infrared regions.

[0021] Ordinary non-mydriatic fundus cameras do not have such dedicated LED illumination devices, and most are equipped with xenon flash lamps. The light emission time of a xenon flash lamp is extremely short, ranging from a few microseconds to a few milliseconds, and the light source spectrum changes rapidly during this time. Therefore, it is not easy to perform hyperspectral imaging using a conventional xenon flash lamp and a tunable bandpass filter.

[0022] The second limitation of applying conventional methods to non-mydriatic fundus imaging is the number of wavelength bands and exposure time. In non-mydriatic imaging without the use of mydriatic drugs, imaging must be completed within approximately 0.2 seconds from the irradiation of illumination light that induces the miosis reflex. In a method of sequentially imaging multiple wavelength bands, the time required to complete imaging of all wavelength bands is (exposure time per wavelength band + switching time) × (number of wavelength bands).

[0023] Therefore, it is necessary to impose limitations on either or both of the exposure time per wavelength band and the number of wavelength bands. For example, if 20 wavelength bands are desired, the exposure time per wavelength band is limited to approximately 10 milliseconds or less. This exposure time is significantly shorter than typical exposure times, and a high-intensity light source is used. The use of a high-intensity light source increases the recovery time from miosis, which increases the time required to image one fundus and then the other, for example, when imaging both left and right funduses. Conversely, if the exposure time is set to 30 milliseconds, the number of wavelength bands that can be imaged is limited to approximately 6 to 7.

[0024] The inventors have conceived an imaging method according to an embodiment of the present disclosure that can acquire images in multiple wavelength bands without any deviation in the imaging position during hyperspectral fundus imaging. The imaging method according to this embodiment enables easy and highly accurate comparison of images between multiple wavelength bands. The imaging method according to this embodiment can use a xenon flash lamp as a light source, making it possible to modify the imaging unit of a conventional non-mydriatic fundus camera equipped with a xenon flash lamp and implement the imaging method according to this embodiment. Because there is no trade-off between the number of wavelength bands and the exposure time, it is possible to ensure the required exposure time or acquire images in the required number of wavelength bands. When irradiating with illumination light containing visible light once, multiple imaging operations, such as reflection hyperspectral imaging and fluorescence imaging (described below), can be performed before pupil constriction occurs due to the irradiation.

[0025] (First embodiment) [Non-mydriatic fundus camera] The configuration of a non-mydriatic fundus camera according to a first embodiment of the present disclosure is shown schematically in Fig. 1. In the accompanying drawings, elements composed of multiple lens groups and lenses with negative power may be represented in the shape of a single lens with positive power. Elements that are necessary for use of the non-mydriatic fundus camera but are not specific to the non-mydriatic fundus camera may be omitted.

[0026] 1 includes an illumination unit 0110, a common optical system 0120, an imaging unit 0130, and a control unit 0140. In this specification, the non-mydriatic fundus camera is also referred to as an "imaging system."

[0027] <Illumination Unit 0110> The illumination unit 0110 includes a light source and an illumination optical system. The light source includes an infrared light source 0111A and a high-brightness broadband light source 0111B.

[0028] The infrared light source 0111A outputs infrared light. This infrared light is used to adjust the focus and imaging position of the optical system without inducing the pupil reflex before starting imaging using light including the visible range. The infrared light source 0111A emits light with a small relative luminosity factor and in a wavelength range that does not induce the pupil reflex, for example, light with a wavelength of 750 nm or more. The infrared light source 0111A may be, for example, an LED (light-emitting diode) with a wavelength of 750 nm or more. Alternatively, the infrared light source 0111A may be a halogen lamp equipped with cut filters for visible light, mid-infrared light, and far-infrared light.

[0029] If the hyperspectral camera is sensitive to the infrared region, the infrared light source 0111A may be used for capturing images in the infrared region using the hyperspectral camera. In this case, a light source that emits light components across the wavelength range to which the hyperspectral camera is sensitive is used. A broadband light source such as a halogen lamp can emit light components across that wavelength range. If the infrared light source 0111 includes multiple infrared LEDs with different wavelengths, the light from the multiple infrared LEDs with different wavelengths may be mixed using a dichroic mirror or the like.

[0030] The high-brightness broadband light-emitting source 0111B is a light source for performing snapshot hyperspectral imaging. The high-brightness broadband light-emitting source 0111B emits light containing components of multiple wavelength bands for snapshot hyperspectral imaging. However, when light irradiation is performed simultaneously with the infrared light source 0111A, the high-brightness broadband light-emitting source 0111B does not need to have components in the wavelength range emitted by the infrared light source 0111A.

[0031] The high-intensity broadband light source 0111B may be, for example, a xenon flash lamp, which is widely used in conventional non-mydriatic fundus cameras. By modifying the camera and the control unit in a non-mydriatic fundus camera equipped with a normal camera, the imaging method according to this embodiment can be implemented.

[0032] The high-brightness broadband light source 0111B may be a broadband LED. Alternatively, it may be configured to mix light from multiple LEDs with different wavelengths. The high-brightness broadband light source 0111B may be, for example, a light source that emits light in a broad band that covers the visible to near-infrared region. The high-brightness broadband light source 0111B may emit light having a wavelength of 400 nm or more and 1000 nm or less, for example.

[0033] In another embodiment, the infrared light source 0111A and the high-brightness broadband light source 0111B may be configured as a common light source, and the emission of light having a wavelength in the infrared region and the emission of light having a broadband wavelength may be switched by inserting or removing a filter and controlling the driving method. In this case, the emission time for emitting light having a wavelength in the infrared region may be longer than the emission time for emitting light having a broadband wavelength.

[0034] The high-brightness broadband light source 0111B emits light in response to a signal from the control unit 0140. The light emission time may be longer or shorter than the exposure time of the snapshot hyperspectral camera described below, or may be the same as the exposure time.

[0035] The illumination unit 0110 irradiates the fundus with light from the light sources 0111A and 0111B. The illumination unit 0110 may include collimator lenses 0112A and 0112B that convert the light from the light sources 0111A and 0111B into collimated light. The illumination unit 0110 may include an optical path converging element 0115, such as a half mirror and / or a dichroic mirror, for converging the light from the multiple light sources 0111A and 0111B. The illumination unit 0110 may also include an optical path adjusting element 0116, which is a mirror that converts the optical path, and a condenser lens 0117 that condenses the collimated light near the rear focal position of the objective lens 0121. In addition to these, the illumination unit 0110 may also include an aperture for adjusting the light intensity and / or irradiation range, a filter for spectrum adjustment, and the like. The illumination unit 0110 may include an optical path switching mechanism, such as a movable mirror, that mechanically changes the optical path, instead of the optical path combining element 0115 .

[0036] The illumination unit 0110 may further include another light source, a collimator lens, and an optical path combining element. For example, if the imaging unit 0130 includes an RGB camera for the visible range in addition to the snapshot hyperspectral camera, a light source for the snapshot hyperspectral camera and a light source for the RGB camera may be provided separately.

[0037] <Common Optical System 0120> The common optical system 0120 includes a beam splitter 0122 and an objective lens 0121. The beam splitter 0122 reflects light from the illumination unit 0110 toward the objective lens 0121, and passes reflected light and / or fluorescence from the fundus that has been collimated by the objective lens 0121. The objective lens 0121 irradiates the fundus with the light from the beam splitter 0122. The objective lens 0121 further forms an image of the fundus on the image plane of the hyperspectral camera using the reflected light and / or fluorescence from the fundus in combination with an imaging lens of the imaging unit. In addition to these, the common optical system 0120 may include a movable mechanism for adjusting the imaging position.

[0038] <Image Capture Unit 0130> The image capture unit 0130 may include a focus lens 0131 and imaging lenses 0132A and 0132B. The image capture unit 0130 may also include an infrared camera 0133A, a snapshot hyperspectral camera 0133B, a half mirror or dichroic mirror 0136, and an optical path adjustment element 0137 such as a mirror.

[0039] The focus lens 0131 may be adjustable in position, optical power, or both, and may form an image of the fundus on the image plane of the infrared camera 0133A and the snapshot hyperspectral camera 0133B together with the objective lens 0121 and the imaging lenses 0132A and 0132B. Instead of using the focus lens 0131, the objective lens 0121 may be made movable so that it functions as a focus lens. Alternatively, the focus lens 0131 may be disposed in the common optical system 0120.

[0040] The infrared camera 0133A is sensitive to the infrared light emitted from the infrared light source 0111A, and is used to check the focus and imaging position of the optical system without inducing the pupillary reflex.

[0041] The half mirror or dichroic mirror 0136 splits the optical path of the light coming from the objective lens 0121 into two so that the light can be captured by the infrared camera 0133A and the snapshot hyperspectral camera 0133B. In the case of a dichroic mirror, the mirror reflects and / or transmits light so that light in the wavelength band captured by the infrared camera 0133A is directed toward the infrared camera 0133A, and reflects and / or transmits light so that light in the wavelength band captured by the snapshot hyperspectral camera 0133B is directed toward the snapshot hyperspectral camera 0133B.

[0042] Instead of the half mirror or dichroic mirror 0136, a movable mirror that mechanically changes the optical path may be used.

[0043] The optical path adjusting element 0137 is used to adjust the optical path. The imaging unit 0130 does not necessarily have to include the optical path adjusting element 0137.

[0044] The image capturing unit 0130 may include a plurality of sets each including a snapshot hyperspectral camera, an imaging lens, and a half mirror or a dichroic mirror, which can expand the wavelength range that can be captured.

[0045] The image capturing unit 0130 may further include a set of an RGB camera, an imaging lens, and a half mirror. In this case, hyperspectral imaging and normal RGB imaging can be performed simultaneously under a single illumination. When the wavelength range that the snapshot hyperspectral camera can capture is limited to a part of the visible range or limited to the infrared range, a visible color image can be obtained along with the hyperspectral image.

[0046] <Control Unit 0140> The control unit 0140 includes a control signal generating device 0141 and a display device 0142. The control signal generating device 0141 generates signals that control the operation of each element of the illumination unit 0110 and the imaging unit 0130. The display device 0142 displays the results of imaging by the imaging unit. The control unit 0140 may also include a memory and a calculation device.

[0047] [Snapshot Hyperspectral Camera] The snapshot hyperspectral camera 0133B is a camera capable of acquiring images in four or more wavelength bands with a single exposure. In other words, the snapshot hyperspectral camera 0133B generates image data including four or more spectral images that correspond one-to-one to the four or more wavelength bands through snapshot imaging, which allows imaging in one shot. The snapshot hyperspectral camera 0133B includes a two-dimensional photoelectric conversion element array and a filter array including at least four or more filter regions with different transmission spectral characteristics.

[0048] The photoelectric conversion element photoelectrically converts light that has passed through the filter array into photoelectric charges. The photoelectric conversion element may include, for example, single-crystal silicon. In this case, the photoelectric conversion element is sensitive to light with a wavelength of approximately 1100 nm or less. Alternatively, the photoelectric conversion element may include indium gallium arsenide. In this case, the photoelectric conversion element is sensitive to light with a wavelength of approximately 1650 nm or less.

[0049] The snapshot hyperspectral camera 0133B has a so-called global shutter function that allows the time when each photoelectric conversion element performs photoelectric conversion to be treated as substantially the same time.

[0050] The shutter operation of the snapshot hyperspectral camera 0133B is performed based on a signal from the control unit 0140.

[0051] The snapshot hyperspectral camera 0133B constructs a hyperspectral image based on the measurement results of the amount of photoelectric charge generated in each photoelectric conversion element and the transmission spectral characteristics of each filter region through which light reaching each photoelectric conversion element passes.

[0052] Snapshot hyperspectral cameras 0133B can be classified into those with filter arrays having high translational symmetry and those with filter arrays having true or pseudo-randomness, depending on the arrangement of their filter regions. The two types differ in the way they construct hyperspectral images. In the first embodiment, a camera with a filter array having high translational symmetry will be described.

[0053] [Filter Array with High Translational Symmetry] A filter array with high translational symmetry is one in which filter regions having substantially the same transmission characteristics as a given filter region are repeatedly arranged at a regular interval. Figure 2 is a diagram showing a schematic diagram of an example of the arrangement of filter regions. In this example, a unit cell 210 includes 16 filter regions arranged in a 4×4 matrix. These 16 filter regions have different transmission characteristics. The unit cells 210 are periodically repeated in both the vertical and horizontal directions.

[0054] In this method, if the horizontal width of a unit cell is X_U and the vertical width is Y_U, filter regions with the same transmission characteristics exist at positions that are an integer multiple of X_U and an integer multiple of Y_U away from a filter region at a certain position on the filter array. Due to their high translational symmetry, the spatial frequencies of filter regions with the same transmission characteristics are unevenly distributed at specific frequencies and their integer multiples.

[0055] The number of filter regions in the horizontal direction and the number of filter regions in the vertical direction included in a unit cell do not necessarily have to be the same, and may be, for example, 3×5 or 4×6.

[0056] In this embodiment, as shown schematically in Figure 3, each filter region has transmission characteristics similar to those of a so-called single bandpass filter, in which the transmittance in a specific wavelength range is significantly high and the transmittance in other wavelength ranges is low. Here, a wavelength range with significantly high transmittance refers to a wavelength range with transmittance equal to or greater than half of the maximum transmittance. A single bandpass filter refers to a filter in which the wavelength range with significantly high transmittance is a single continuous wavelength range. While Figure 3 illustrates transmission characteristics with multiple transmittance maxima, transmission characteristics with only a single transmittance maxima may also be used.

[0057] In this method, substantially identical filter regions appear periodically in space. When the width and height of the photoelectric conversion element are substantially the same as the width and height of the filter region, the transmission characteristics of the filter region and the photoelectric conversion element correspond one-to-one. Therefore, by collecting data on the photocharges of the light source conversion elements corresponding to the same filter region, an image of the wavelength band in which the filter region has significantly higher transmittance can be obtained.

[0058] If the filter array includes n filter regions with different transmission characteristics, the snapshot hyperspectral camera 0133B can generate n images corresponding to n wavelength bands.

[0059] In some cases, pixel interpolation or other computational processing may be performed to improve spatial resolution, as disclosed, for example, in "A CMOS-Compatible, Integrated Approach to Hyper- and Multispectral Imaging" (DOI: 10.1109 / IEDM.2014.7047025).

[0060] The advantage of this type of snapshot hyperspectral camera is that the processing load required to acquire a hyperspectral image from the measurement of photoelectric charge is low, making it suitable for applications such as irradiating a wide band of infrared light in a range that does not induce the pupil reflex and displaying an infrared hyperspectral image in real time.

[0061] The snapshot hyperspectral camera 0133B may include a filter array including a filter region that transmits light with wavelengths having low luminous efficacy. As described above, light with wavelengths having low luminous efficacy includes, for example, light in a wavelength range of 750 nm or greater. This enables the snapshot hyperspectral camera 0133B to capture images in this wavelength range, allowing the non-mydriatic fundus camera to share a camera for adjusting the focus and imaging position and a camera for fundus imaging. In other words, the non-mydriatic fundus camera can adjust the focus and imaging position using the snapshot hyperspectral camera 0133B without using the infrared camera 0133A.

[0062] Second Embodiment A non-mydriatic fundus camera according to a second embodiment of the present disclosure includes a compressed sensing hyperspectral camera. The compressed sensing hyperspectral camera is a snapshot hyperspectral camera that includes a filter array having true or pseudo-random properties.

[0063] Although an additional configuration for performing fluorescence imaging will be described here, this additional configuration can be omitted if fluorescence imaging is not performed. The additional configuration for performing fluorescence imaging may be applied to the non-mydriatic fundus camera according to the first embodiment.

[0064] The configuration of the non-mydriatic fundus camera according to this embodiment is shown schematically in Fig. 4. Note that a description of elements common to the first embodiment will be omitted.

[0065] The non-mydriatic fundus camera shown in FIG. 4 includes an illumination unit 0410, a common optical system 0420, an imaging unit 0430, and a control unit 0440.

[0066] <Illumination Unit 0410> The illumination unit 0410 includes a light source and an illumination optical system. The light source includes an infrared light source 0111A, a high-brightness broadband light source 0111B, and a fluorescence excitation light source 0111C. The illumination unit 0410 further includes a collimator lens 0112C for collimating the excitation light, and an optical path combining element 0415.

[0067] The fluorescence excitation light source 0111C emits excitation light that includes wavelength components effective for exciting the fluorescence to be imaged, but that does not substantially include wavelength components to be imaged.

[0068] The wavelengths effective for exciting fluorescence and the resulting fluorescence wavelengths effective for imaging vary depending on the fluorescent substance. Fluorescence imaging can be performed by administering a fluorescent agent to a subject and then imaging the fluorescence of fluorescent substances inherent in the subject's tissues, known as autofluorescence. The latter method is called autofluorescence imaging.

[0069] The former method must be performed separately from the imaging to acquire a reflected image because the drug also affects the reflected image. On the other hand, the latter method is unaffected by the drug, so the imaging to acquire an autofluorescence image can be performed immediately after the imaging to acquire a reflected image. As long as the total imaging time for acquiring the reflected image, the imaging time for acquiring the autofluorescence image, and the switching time between the two are within approximately 0.2 seconds, which is limited by the miosis reflex, both images can be captured in a single, non-mydriatic state. The imaging method will be described in detail later.

[0070] Lipofuscin in retinal pigment epithelial cells and melanolipofuscin, in which melanin is covered with lipofuscin, are known to be autofluorescent substances. Lipofuscin autofluorescence imaging is known to be effective in examining diseases such as retinal pigment epithelial atrophy. The autofluorescence spectrum of lipofuscin can change due to disease or aging. Therefore, analyzing the fluorescence spectrum of lipofuscin makes it possible to determine disease or aging in the subject's fundus. Hyperspectral imaging can detect changes in the fluorescence spectrum more sensitively than monochrome and RGB imaging.

[0071] Lipofuscin can be excited to fluoresce with light having a wavelength in the range of about 400 nm to 600 nm, and the fluorescence has components in the wavelength range of about 500 nm to 800 nm. Melanolipofuscin can be excited to fluoresce with light having a wavelength of about 780 nm, and the fluorescence has components in the wavelength range of about 800 nm or more.

[0072] Fluorescent components with wavelengths shorter than the excitation wavelength are usually not observed.

[0073] The fluorescence excitation light source 0111C for obtaining an autofluorescence image of lipofuscin can be a light source that emits light having components in the wavelength range of 400 nm to 600 nm, for example, an LED having a light source spectrum with a peak wavelength of 420 nm. The fluorescence excitation light source 0111C may be equipped with a short-pass or band-pass filter to remove components of the excitation light whose wavelength range overlaps with the fluorescence spectrum.

[0074] The common optical system 0420 is the same as the common optical system 0120 in the first embodiment, and therefore a description thereof will be omitted.

[0075] The imaging unit 0430 includes a focus lens 0131, an imaging lens 0432, a compressed sensing hyperspectral camera 0433, an excitation cut filter 0434, and an optical path adjustment element 0137 for adjusting the optical path.

[0076] In this embodiment, the compressed sensing hyperspectral camera 0433 is sensitive to infrared light emitted from the infrared light source 0111A, and is used to check the focus and imaging position of the optical system without inducing the pupil reflex. Of course, as in the first embodiment, an infrared camera and a dedicated imaging lens may also be provided.

[0077] The excitation cutoff filter 0434 is a filter that blocks light emitted from the fluorescence excitation light source 0111C during fluorescence imaging and transmits fluorescence emitted by the imaging target. If it is not necessary to capture the wavelength band of the excitation light in reflection imaging, for example, if the excitation light has a wavelength of 420 nm and the wavelength range for reflection imaging is 450 nm to 1000 nm, a long-pass filter that blocks light with a wavelength of less than 450 nm may be always placed on the optical path.

[0078] On the other hand, if the wavelength of the excitation light overlaps part of the wavelength range of the reflection imaging, the excitation cutoff filter 0434 may be inserted into the optical path when performing fluorescence imaging, and removed from the optical path when performing reflection imaging. Such a filter can be inserted or removed using an electric filter wheel or electric slider that can automatically perform the insertion or removal in response to a signal. By configuring the filter to be inserted or removed automatically in response to a signal, it becomes possible to switch between reflection imaging and fluorescence imaging during the same time that the pupil is dilated.

[0079] [Compressed Sensing Hyperspectral Camera] The compressed sensing hyperspectral camera 0433 is a camera that has the function of utilizing the sparsity of the subject and mask to perform computational reconstruction of hyperspectral images of multiple wavelength bands included in the wavelength range of the imaged object from a single captured image called a compressed image. The compressed image contains information from four or more wavelength bands. The compressed sensing hyperspectral camera 0433 generates a compressed image by taking snapshots.

[0080] The compressed sensing hyperspectral camera 0433, like the snapshot hyperspectral camera 0133B described in the first embodiment, includes a two-dimensional photoelectric conversion element array and a filter array including at least four or more filter regions with different transmission spectral characteristics. The photoelectric conversion element array is the same as that described in the first embodiment.

[0081] On the other hand, a filter array can be truly or pseudo-random. "True or pseudo-random" means that the spatial distribution of transmittance for a certain wavelength band is completely random, or that its spatial frequency components include not only integer multiples of a single frequency but also multiple components that are not integer multiples. A filter array is sometimes called a coded mask.

[0082] An example of the spatial distribution of transmittance for a certain wavelength band and the spatial distribution of transmittance for another wavelength band is shown in Figure 5. Here, darker colors represent lower transmittance, and lighter colors represent higher transmittance. W1 represents a first wavelength band included in the plurality of wavelength bands, W2 represents a second wavelength band included in the plurality of wavelength bands, and Wi represents the i-th wavelength band included in the plurality of wavelength bands. As shown in Figure 5, the spatial distribution of transmittance differs depending on the wavelength band.

[0083] The filter array includes a first filter region exhibiting a first transmission spectral characteristic, through an nth filter region exhibiting an nth transmission spectral characteristic, where n is an integer equal to or greater than 4. The first transmission spectral characteristic, through an nth transmission spectral characteristic are different from each other.

[0084] 6A shows an example of a first transmission spectrum of the first filter region, and FIG. 6B shows an example of a second transmission spectrum of the second filter region, where the transmission spectrum characteristics of the first transmission spectrum are different from the transmission spectrum characteristics of the second transmission spectrum.

[0085] The first transmission spectral characteristic has a plurality of maxima in the wavelength range of the imaging target. The first transmission spectral characteristic has periodic multi-peaks in the wavelength direction. The first transmission spectral characteristic may have a plurality of maxima appearing randomly in the wavelength range of the imaging target.

[0086] The second transmission spectral characteristic has a plurality of maxima in the wavelength range of the imaging target. The second transmission spectral characteristic has periodic multi-peaks in the wavelength direction. The second transmission spectral characteristic may have a plurality of maxima appearing randomly in the wavelength range of the imaging target.

[0087] The filter regions included in the filter array may be Fabry-Perot filters having resonant structures.

[0088] The coded mask included in the compressed sensing hyperspectral camera 0433 does not have to have filter regions arranged in an array. The coded mask may be an optical element including a metalens using an optical metasurface. The optical element may have multiple regions with different transmission spectra.

[0089] In this method, each photoelectric conversion element detects light affected by a corresponding filter region among a plurality of filter regions with different transmission spectra. An image affected by such a coding mask is called a compressed image.

[0090] By performing an estimation operation using the two pieces of information, namely, the compressed image and the transmission spectrum data for each filter region of the encoding mask, and assuming the sparsity of the object, it is possible to estimate an image in each wavelength band. This estimation operation process is called sparse reconstruction, and the estimated image is called a restored image.

[0091] Details of this technology are disclosed in, for example, International Publication No. 2022 / 244645.

[0092] In the compressed sensing hyperspectral camera 0433, each photoelectric conversion element simultaneously detects light in multiple wavelength bands, resulting in high light utilization efficiency. This results in sensitivity several times higher than that of a snapshot hyperspectral camera equipped with a filter array with high translational symmetry. This means that the (illumination intensity) × (exposure time) required to acquire a comparable image can be reduced to a fraction of the original value.

[0093] This feature allows for the illumination intensity to be reduced, enabling imaging with weaker illumination. Weaker illumination shortens the time it takes for the pupil to dilate sufficiently to allow imaging of the other fundus, for example, when imaging the left and right fundus sequentially. As a result, imaging of both eyes can be completed in a shorter time.

[0094] By allocating exposure time based on this characteristic, multiple imaging sessions can be performed within the approximately 0.2 seconds between the irradiation of illumination light containing visible light and pupil contraction. For example, after performing reflectance hyperspectral imaging in 0.05 seconds, it is possible to switch from reflectance hyperspectral imaging mode to autofluorescence imaging mode over 0.05 seconds, and then perform fluorescence imaging in 0.1 seconds.

[0095] In the compressed sensing hyperspectral camera 0433, each photoelectric conversion element simultaneously detects light in multiple wavelength bands. Therefore, the compressed sensing hyperspectral camera 0433 functions as a pseudo-monochrome camera at the stage of compressed images before restoration. Even a hyperspectral camera whose primary purpose is to acquire hyperspectral images in the visible range can be configured to acquire monochrome images in the infrared range. Therefore, a compressed sensing hyperspectral camera configured to be sensitive to infrared light emitted from the infrared light source 0111A can be used to confirm the focus and imaging position of an optical system without inducing the pupil reflex.

[0096] Although the compressed sensing hyperspectral camera 0433 is naturally capable of sparsely reconstructing hyperspectral images in the infrared region, there is a possibility of a time lag between exposure and screen display during video capture during the sparse reconstruction calculation process. Displaying the compressed image as is can shorten this time, allowing for quick adjustment of the focus and imaging position.

[0097] In the compressed sensing hyperspectral camera 0433, the number of wavelength bands in the restored image can be freely selected during sparse reconstruction. Such processing is disclosed, for example, in WO 2023 / 282069.

[0098] The number of wavelength bands is not limited by the number of filter regions with different transmission spectral characteristics. Therefore, the number of wavelength bands can be increased without sacrificing resolution. The compressed sensing hyperspectral camera 0433 can perform different sparse reconstructions for each disease to be diagnosed. The compressed sensing hyperspectral camera 0433 can perform different sparse reconstructions for each biological element to be diagnosed. The compressed sensing hyperspectral camera 0433 may store information indicating wavelength bands suitable for diagnosing specific diseases or biological elements in a storage device. The compressed sensing hyperspectral camera 0433 may determine or change the wavelength bands for sparse reconstruction based on the information.

[0099] <Control Unit 0440> The control unit 0440 includes a control signal generating device 0141, a display device 0142, and a memory 0443. The control signal generating device 0141 generates signals that control the operation of each element of the illumination unit 0410 and the imaging unit 0430. The display device 0142 displays the results of imaging by the imaging unit.

[0100] The memory 0443 stores compressed images acquired by the compressed sensing hyperspectral camera 0433. The memory 0443 may store matrix data that reflects the spatial distribution of the transmission spectrum of the coding mask.

[0101] When a camera that acquires compressed images but does not restore the hyperspectral images is used instead of the compressed sensing hyperspectral camera 0433, the control unit 0440 may include a calculation device 0445 for performing restoration processing. The calculation device 0445 may execute processing to generate four or more spectral images that correspond one-to-one to four or more wavelength bands based on the compressed image and the matrix data.

[0102] [Method for Performing Reflection Hyperspectral Imaging Without Fluorescence Imaging] A method for imaging a non-mydriatic fundus using a non-mydriatic fundus camera according to this embodiment will be described. First, a case where reflection hyperspectral imaging is performed without fluorescence imaging using the non-mydriatic fundus camera according to the first embodiment will be described.

[0103] 7 is a flowchart showing a process for performing reflection hyperspectral imaging of a non-mydriatic fundus using the non-mydriatic fundus camera according to the first embodiment. In the following description, infrared light is also referred to as "infrared light." The flowchart shown in FIG. 7 includes steps S1 to S5.

[0104] <S1: Emit infrared light> The infrared light source 0111A is caused to emit infrared light. As a result, the fundus of the subject whose pupils have naturally dilated while waiting in a dark place is irradiated with the infrared light emitted from the infrared light source 0111A, and video images are captured by the infrared camera 0133A.

[0105] <S2: Adjusting the focus> A moving image obtained by the infrared camera 0133A is displayed on a monitor, and the focus is adjusted based on the displayed moving image. Instead of adjusting the focus, the imaging position may be adjusted, or both the focus and the imaging position may be adjusted. It is not necessary to display the moving image. The control unit 0140 may adjust the focus and the imaging position, or the user operating the non-mydriatic fundus camera may manually adjust the focus and the imaging position based on the displayed moving image.

[0106] The light emission of the infrared light source 0111A may be stopped after the focus and imaging position adjustments are completed. Alternatively, the light emission may not be stopped if the snapshot hyperspectral camera 0133B is not sensitive to the illumination wavelength of the infrared light source 0111A. Alternatively, if the infrared light emitted from the infrared light source 0111A is also used for snapshot hyperspectral imaging, the light emission by the infrared light source 0111A may be continued until the exposure of the hyperspectral camera is completed. Alternatively, the light emission may be stopped once and then turned on again during snapshot hyperspectral imaging.

[0107] <S3: Start exposure> The photographer or the device decides to start imaging, and generates an imaging start signal from the control device and sends it to the snapshot hyperspectral camera 0133B.

[0108] The snapshot hyperspectral camera 0133B begins exposure.

[0109] When the snapshot hyperspectral camera 0133B described in embodiment 1 is used as the snapshot hyperspectral camera, the snapshot hyperspectral camera 0133B generates image data including information on four or more wavelength bands by exposure, and the image data includes four or more spectral images corresponding to each of the four or more wavelength bands.

[0110] When the compressed sensing hyperspectral camera 0433 described in embodiment 2 is used as the snapshot hyperspectral camera, the compressed sensing hyperspectral camera 0433 generates image data including information of multiple wavelength bands by exposure, and the image data includes a compressed image in which the information of the multiple wavelength bands is compressed into a single image. The compressed image has multiple pixels, and information of multiple wavelength bands is superimposed on each of the multiple pixels. The multiple wavelength bands may be, for example, four or more wavelength bands.

[0111] <S4: Emitting Broadband Light> The high-brightness broadband light source 0111B is caused to emit broadband light within the period until the exposure of the snapshot hyperspectral camera 0133B is completed.

[0112] <S5: Output Image Data> The hyperspectral camera completes exposure, generates image data including information on four or more wavelength bands, and outputs the image data to the memory 0443 or the like.

[0113] The generated image data is generated by detecting, with the hyperspectral camera, return light from the subject's eye due to the broadband light emitted from the high-brightness broadband light source 0111B in S4. In this specification, the image data may be referred to as "first image data."

[0114] The snapshot hyperspectral camera 0133B may be configured to repeatedly generate image data, and the high-brightness broadband light source 0111B may be configured to emit light during an exposure period for generating any of the image data.

[0115] FIG. 8 shows a timing chart corresponding to the light emission period of the infrared light source 0111A, the light emission period of the high brightness broadband light source 0111B, and the exposure period of the snapshot hyperspectral camera 0133B.

[0116] After the infrared radiation in the darkroom has finished, that is, after the setting of the photographing conditions such as the focus of the snapshot hyperspectral camera 0133B has been completed, exposure of the snapshot hyperspectral camera 0133B begins.

[0117] The broadband light is emitted from the high-brightness broadband light source 0111B so as to overlap with the exposure period of the snapshot hyperspectral camera 0133B. As shown in Figure 8, the light emission period of the high-brightness broadband light source 0111B falls after the start of exposure of the snapshot hyperspectral camera 0133B and before the end of exposure.

[0118] Here, the irradiation of the broadband light initiates pupil constriction in the subject's eye. The snapshot hyperspectral camera 0133B stops the exposure before the pupil constriction ends. The camera then outputs image data generated based on the exposure. This allows hyperspectral imaging of the fundus without mydriasis while minimizing the burden on the subject caused by multiple irradiations of broadband light.

[0119] Because the snapshot hyperspectral camera 0133B has a global shutter function, even if the light source spectrum of the high-brightness broadband light source 0111B changes during the exposure period, the image capture is affected by the time integral value during the exposure period. In other words, the change in the light source spectrum does not affect the image comparison between wavelength bands.

[0120] Of course, since the images of each wavelength band are acquired simultaneously, there is no deviation in the imaging position.

[0121] An advantage of making the light-emitting period of the high-brightness broadband light-emitting source 0111B shorter than the exposure period is that the influence of eye movement on imaging can be reduced. Even if there is exposure during the period when the high-brightness broadband light-emitting source 0111B is not emitting light, no photocharge is generated. Therefore, it is possible to obtain the same effect as if there was no exposure during the period when the high-brightness broadband light-emitting source 0111B is not emitting light, and exposure was performed during the period when the high-brightness broadband light-emitting source 0111B is emitting light. A xenon flash lamp is particularly effective because it can emit light with high brightness for a short period of time.

[0122] Another advantage is that the accuracy requirements for synchronization between broadband light emission and exposure can be reduced. If the entire broadband light emission period is included within the exposure period, any deviation in the time of broadband light emission within that range will not affect the imaging results. On the other hand, if part of the broadband light emission period is included in the exposure period and the remaining part is outside the exposure period, the imaging results will vary depending on how much of the broadband light emission period is included in the exposure period.

[0123] The emission of the broadband light and the exposure are controlled by the control signal generator 0141, but jitter can cause uncertainty in their synchronization. As described above, by making the light emission period of the high-brightness broadband light-emitting source 0111B shorter than the exposure period, the influence of jitter on the imaging results can be eliminated.

[0124] [Method for successively performing reflective hyperspectral imaging and fluorescent imaging] A case where reflective hyperspectral imaging and fluorescent imaging are successively performed using the non-mydriatic fundus camera according to the second embodiment will be described.

[0125] 9 is a flowchart showing a process for performing reflective hyperspectral imaging and fluorescence imaging of a non-mydriatic fundus using a non-mydriatic fundus camera according to the second embodiment. The flowchart shown in FIG. 9 includes steps S11 to S18. Steps S11, S12, S13, S14, and S15 shown in FIG. 9 are the same as steps S1, S2, S3, S4, and S5 shown in FIG. 7, respectively, and therefore will not be described here.

[0126] <S16: Emitting Excitation Light> After the compressed sensing hyperspectral camera 0433 has completed exposure for generating the first image data, the fluorescence excitation light source 0111C is caused to emit excitation light.

[0127] <S17: Start of exposure> After the fluorescence excitation light source 0111C starts emitting light, the compressed sensing hyperspectral camera 0433 starts exposure.

[0128] The exposure in S17 is for detecting autofluorescence caused by the light emitted from the fluorescence excitation light source 0111C.

[0129] The fluorescence excitation light source 0111C and the compressed sensing hyperspectral camera 0433 may be controlled so that the emission period of the fluorescence excitation light source 0111C and the exposure period of the compressed sensing hyperspectral camera 0433 overlap. The emission period of the fluorescence excitation light source 0111C and the exposure period of the compressed sensing hyperspectral camera 0433 may only partially overlap. Exposure of the compressed sensing hyperspectral camera 0433 may start simultaneously with emission of light from the fluorescence excitation light source 0111C. Exposure of the compressed sensing hyperspectral camera 0433 may start before emission of light from the fluorescence excitation light source 0111C begins.

[0130] When it is desired to capture an image of a wavelength band that is not transmitted by the excitation cut filter 0434, the excitation cut filter 0434 may be retracted from the optical path.

[0131] <S18: Output second image data> The compressed sensing hyperspectral camera 0433 finishes exposure, generates image data including information of four or more wavelength bands, and outputs the image data to the memory 0443 or the like.

[0132] The image data is generated by detecting autofluorescence from at least a portion of the subject's fundus due to the excitation light emitted in S16 using the compressed sensing hyperspectral camera 0433. In this specification, this image data is also referred to as "second image data."

[0133] Alternatively, the compressed sensing hyperspectral camera 0433 may be configured to repeatedly generate image data, and the fluorescence excitation light source 0111C may be configured to emit light during an exposure period for generating any of the image data.

[0134] After the compressed sensing hyperspectral camera 0433 outputs the second image data, the compressed sensing hyperspectral camera 0433 may perform restoration processing based on the first image data to generate four or more spectral images that correspond one-to-one to the four or more wavelength bands. The restoration processing based on the first image data may be performed after the output of the first image data and before the output of the second image data. The calculation device 0445 may perform restoration processing based on the first image data to generate four or more spectral images that correspond one-to-one to the four or more wavelength bands.

[0135] After the compressive sensing hyperspectral camera 0433 outputs the second image data, the compressive sensing hyperspectral camera 0433 may execute a restoration process based on the second image data to generate four or more spectral images that correspond one-to-one to the four or more wavelength bands. The calculation device 0445 may execute a restoration process based on the second image data to generate four or more spectral images that correspond one-to-one to the four or more wavelength bands.

[0136] The set of steps for generating the first image data, including S13, S14, and S15, may be performed in a reverse order with the set of steps for generating the second image data, including S16, S17, and S18.

[0137] 10 shows a timing chart corresponding to the light emission period of the infrared light source 0111A, the light emission period of the high-brightness broadband light emission source 0111B, the light emission period of the fluorescence excitation light source 0111C, and the exposure period of the compressed sensing hyperspectral camera 0433. Description of content that overlaps with that of FIG. 8 will be omitted.

[0138] After the compressed sensing hyperspectral camera 0433 has completed exposure for generating the first image data, the fluorescence excitation light source 0111C starts emitting light.

[0139] The compressed sensing hyperspectral camera 0433 starts exposure so that the light emission period of the fluorescence excitation light source 0111C overlaps with the exposure period of the compressed sensing hyperspectral camera 0433. As described above, the timing at which the compressed sensing hyperspectral camera 0433 starts exposure may be simultaneous with or different from the timing at which the fluorescence excitation light source 0111C starts emitting light.

[0140] The irradiation of the broadband light initiates pupil constriction in the subject's eye. The compressed sensing hyperspectral camera 0433 stops the exposure, which started to overlap with the light emission period of the fluorescence excitation light source 0111C, before the pupil constriction ends. The compressed sensing hyperspectral camera 0433 then outputs second image data generated based on the exposure. This makes it possible to perform non-mydriatic hyperspectral imaging of the fundus in response to the irradiation of the broadband light and the excitation light, while minimizing the burden on the subject caused by multiple irradiations with the broadband light and the excitation light.

[0141] As described above, irradiation with excitation light may be performed before irradiation with broadband light. In this case, pupil constriction of the subject begins due to irradiation with excitation light. Even in this case, by completing the exposure for generating the first image data and the exposure for generating the second image data before pupil constriction ends, it is possible to perform non-mydriatic hyperspectral imaging of the fundus in response to irradiation with broadband light and excitation light while minimizing burden on the subject.

[0142] Even when fluorescence imaging is performed by irradiating excitation light without irradiating broadband light, the exposure for generating the second image data may be terminated before pupil constriction due to the irradiation of excitation light occurs.

[0143] [Analysis Based on Fluorescence Spectrum] An example of analysis based on fluorescence spectrum will be described. In the following description, the analysis is performed by the calculation device 0445. The analysis may also be performed by a device other than that included in the non-mydriatic fundus camera.

[0144] The calculation device 0445 performs analysis based on the fluorescence spectrum using a plurality of spectral images acquired by fluorescence imaging, which correspond one-to-one to a plurality of wavelength bands.

[0145] The analysis can be, for example, a method for distinguishing between multiple fluorescence spectra. Two methods are described below.

[0146] The first method is a method of estimating a wavelength band including a peak wavelength.

[0147] When images of n wavelength bands are acquired using a hyperspectral camera, n types of intensity data are obtained from each pixel. By identifying the wavelength band showing the highest intensity for each pixel, the wavelength range including the peak wavelength of the fluorescence spectrum at that pixel can be determined. With regard to lipofuscin, the peak wavelengths of the normal group and the aging group are different, so the two can be distinguished from each other using the wavelength band including the peak wavelength.

[0148] The above-described method for estimating a wavelength band including a peak wavelength may include the following steps (step 1) to (step 4).

[0149] (Step 1) The compressed sensing hyperspectral camera 0433 or the computing device 0445 detects the wavelength band １ The corresponding image １ , ~, wavelength band ｎ The corresponding image ｎ Generate an image １ is multiple pixel values,,,,image ｎ contains multiple pixel values.

[0150] image １ The multiple pixel values ​​contained in １ The pixel value I1(1,1) of the pixel located at the coordinate (1,1) in the image １ is the pixel value I1(j,k) of the pixel located at coordinates (j,k) in the image ｎ The multiple pixel values ​​contained in ｎ The pixel value In(1,1) of the pixel located at the coordinate (1,1) in the image １ is the pixel value In(j,k) of the pixel located at coordinates (j,k) in

[0151] (Step 2) The calculation device 0445 determines Imax(1,1), which is the maximum value among I1(1,1), ~, In(1,1), and Imax(j,k), which is the maximum value among ~, I1(j,k), ~, In(j,k).

[0152] (Step 3) The calculation device 0445 determines the wavelength bands λ11, . . . corresponding to Imax(1,1), and the wavelength bands λjk corresponding to Imax(j,k).

[0153] For example, if Imax(1,1) is Ia(1,1), the calculation device 0445 determines that the wavelength band λ11 is wavelength band a (1≦a≦n).

[0154] (Step 4) The calculation device 0445 determines the wavelength range including the peak wavelength based on the wavelength bands λ11, . . . , and wavelength band λjk.

[0155] For example, the calculation device 0445 determines that the wavelength band that is most frequently included in the wavelength bands λ11, . . . , λjk is the wavelength range that includes the peak wavelength.

[0156] With regard to lipofuscin, the peak wavelength of the normal group and the peak wavelength of the aged group are different from each other, so the calculation device 0445 can distinguish between the two groups based on the wavelength range including the peak wavelength.

[0157] Taking into account the influence of noise, etc., it is also possible to take the average value of the intensities of multiple adjacent sections, such as three sections, and search for the wavelength section where this is the highest, rather than simply searching for the wavelength range that shows the strongest intensity.

[0158] In this method of searching for peak wavelengths, wavelength bands are selected so that the peak wavelengths of the two fluorescence spectra to be distinguished are included in different wavelength bands. For example, with regard to lipofuscin, the peak wavelength of the normal group is in the wavelength range of 563 nm to 564 nm, while the peak position of the aging group is in the wavelength range of 528 nm to 563 nm. To make these two groups distinguishable, hyperspectral imaging is performed so that the multiple wavelength bands include the following first and second wavelength bands. The first wavelength band includes at least a portion of the wavelength range of 563 nm to 564 nm, but does not include the wavelength range of 528 nm to 563 nm. The second wavelength band includes at least a portion of the wavelength range of 528 nm to 563 nm, but does not include the wavelength range of 563 nm to 564 nm.

[0159] In the first embodiment, "including the wavelength range" means that the transmission characteristics of the filter region exhibit more than half of the maximum transmittance in the wavelength range, and "excluding the wavelength range" means that the transmission characteristics of the filter region exhibit wavelengths that are less than half of the maximum transmittance in the wavelength range.

[0160] The second method is by difference or comparison.

[0161] When performing the calculation (intensity of the first wavelength band) - (intensity of the second wavelength band) for each pixel for images of the first and second wavelength bands, if the intensity of the first wavelength band is high, the value will be positive; if the intensity is the same, the value will be zero; and if the intensity of the second wavelength band is high, the value will be negative.

[0162] Alternatively, when performing the calculation (intensity of the first wavelength band) ÷ (intensity of the second wavelength band) for each pixel for images of the first and second wavelength bands, if the intensity of the first wavelength band is high, the value will be greater than 1; if the intensity is the same, the value will be 1; and if the intensity of the second wavelength band is high, the value will be less than 1.

[0163] When a fluorescence spectrum is symmetrically distributed around a single peak wavelength, the intensities of two wavelength bands will be the same if they are equidistant from the peak wavelength in wavelength space. If one of the two wavelength bands is closer to the peak position in wavelength space than the other, the intensity of the one wavelength band will be higher than the intensity of the other wavelength band.

[0164] For example, examples of fluorescence spectra for the normal group and the aging group are shown schematically in FIG. 11 . The solid and dashed lines represent the fluorescence spectra for the normal group and the aging group, respectively. As shown in FIG. 11 , in the fluorescence spectrum of the aging group, the wavelength band B1 from 500 nm to 525 nm and the wavelength band B2 from 525 nm to 550 nm are approximately equidistant from the peak wavelength, so the intensities of wavelength band B1 and wavelength band B2 are approximately equal. In contrast, in the fluorescence spectrum of the normal group, wavelength band B2 is closer to the peak wavelength than wavelength band B1, so the intensity of wavelength band B2 is higher than the intensity of wavelength band B1. Therefore, by performing the above calculations and judgments, the two can be distinguished.

[0165] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0166] [Technology 1] A method comprising: illuminating a subject's eye in a darkroom with a first light; and generating first image data indicating the fundus of the subject, the first image data including information of four or more wavelength bands, wherein the first image data is generated by having a camera take a snapshot of the subject's eye before pupil constriction of the eye caused by the illumination of the first light is completed.

[0167] This method makes it possible to acquire images of multiple wavelength bands without shifting the imaging position in hyperspectral imaging of the fundus.

[0168] [Technology 2] The method according to Technology 1, wherein the first image data includes a compressed image in which information of the four or more wavelength bands is compressed.

[0169] This method allows the camera to be, for example, a compressed sensing hyperspectral camera.

[0170] [Technology 3] The method according to Technology 1, wherein the first image data includes four or more spectral images that correspond one-to-one to the four or more wavelength bands.

[0171] This method allows the camera to be, for example, a snapshot hyperspectral camera with a filter array that has high translational symmetry.

[0172] [Technology 4] The method according to any one of Techniques 1 to 3, further comprising irradiating the subject's eye with a second light including infrared light for adjusting the focus position of the camera.

[0173] This method allows the camera to be focused without inducing the pupillary constriction.

[0174] [Technology 5] The method according to Technology 1, wherein the first light includes excitation light, and the first image data is generated by capturing fluorescence caused by the excitation light with the camera.

[0175] This method allows for fluorescence imaging.

[0176] [Technology 6] The method according to Technology 5, further comprising: estimating a peak wavelength of the fluorescence from at least a part of the region of the fundus based on the first image data.

[0177] This method allows the peak wavelength of the fluorescence to be estimated.

[0178] [Technology 7] The method according to Technology 6, wherein the peak wavelength is estimated by comparing information of two or more wavelength bands among the four or more wavelength bands included in the first image data.

[0179] This method allows the peak wavelength of the fluorescence to be easily estimated.

[0180] [Technology 8] The method according to Technology 1, further comprising: irradiating the eye of the subject with a third light including excitation light; and generating second image data based on fluorescence caused by the third light, wherein the first image data and the second image data are generated by causing the camera to capture a snapshot before pupil constriction of the eye caused by irradiation with the first light or the third light ends.

[0181] In this way, the first and second images can be acquired before the pupil of the eye has finished contracting due to the light exposure.

[0182] [Technology 9] An imaging system comprising: a light source that irradiates a subject's eye in a darkroom with a first light; and a camera that generates first image data that indicates the fundus of the subject and includes information in four or more wavelength bands, wherein the camera generates the first image data by capturing a snapshot of the subject's eye before pupil constriction of the eye caused by the irradiation of the first light ends.

[0183] This imaging system can acquire images in multiple wavelength bands without any deviation in the imaging position during hyperspectral imaging of the fundus.

[0184] [Technology 10] The imaging system according to Technology 9, wherein the camera includes an optical element including four or more regions having mutually different transmission spectra.

[0185] In this imaging system, the camera can be, for example, a compressed sensing hyperspectral camera or a snapshot hyperspectral camera with a filter array that has high translational symmetry.

[0186] [Technology 11] The imaging system according to Technology 10, wherein the transmission spectrum of each of the four or more regions has a plurality of maxima in a wavelength range of an imaging target, and the first image data includes a compressed image in which information of a plurality of wavelength bands is compressed.

[0187] In this imaging system, for example, a compressed sensing hyperspectral camera can be used as the camera.

[0188] [Technology 12] The imaging system described in Technology 10, wherein the peak wavelengths of the transmission spectra of the four or more regions are different from each other, and the first image data includes four or more spectral images that correspond one-to-one to the four or more wavelength bands.

[0189] In this imaging system, the camera can be, for example, a snapshot hyperspectral camera with a filter array that has high translational symmetry.

[0190] (Other 1) A method for generating hyperspectral image data from data of one compressed image may be as follows: Data of one compressed image is written as g.

[0191] Hyperspectral images are images of wavelength bands １ The corresponding image １ , wavelength band ２ The corresponding image ２ , ..., wavelength band Ｎ The corresponding image Ｎ The hyperspectral image data f includes the image １ Data f １ ,image ２ Data f ２ ,···,image Ｎ Data f Ｎ Includes:

[0192] Compressed images, images １ ,image ２ ,···,image Ｎ The horizontal direction of each image is the x direction, and the vertical direction of the image is the y direction. １ ,image ２ ,···,image Ｎ The number of pixels in the x direction is m, and the number of pixels in the y direction is n.

[0193] Data g, Data f １ , data f ２ , ..., data f Ｎ Each of the elements has m×n pixel values ​​that correspond one-to-one to the m×n pixels. The data f has m×n×N pixel values.

[0194] The compressed image data g obtained by encoding and multiplexing using the filter array includes m×n pixel values ​​that correspond one-to-one to the m×n pixels. The data g can be expressed by the following equation (1).

[0195]

[0196] In equation (1), f represents the hyperspectral image data expressed as a one-dimensional vector. １ , f ２ , ..., f Ｎ Each of these has m×n elements. Therefore, the vector on the right side is a one-dimensional vector with m×n×N rows and 1 column. In equation (1), the compressed image data g is converted into a one-dimensional vector with m×n rows and 1 column. The matrix H is expressed by converting each component f of the vector f １ , f ２ , ..., f Ｎ represents a transformation in which each wavelength band is encoded with different encoding information, intensity-modulated, and then added together. Therefore, H is an m×n row and m×n×N column matrix. Equation (1) can also be expressed as follows:

[0197] g = (pg １１ ...pg １ｎ ...pg ｍ１ ...pg ｍｎ ) Ｔ = H(f １ ...f Ｎ ) Ｔ Here, pg ｉｊ represents the pixel value at the i-th row and j-th column of the compressed image.

[0198] Given a vector g and a matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements m×n×N of the desired data f is greater than the number of elements m×n of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, a solution is found using a compressed sensing technique, taking advantage of the sparsity of the image contained in the data f. Specifically, the desired data f is estimated by solving the following equation (2).

[0199]

[0200] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means that f that minimizes the sum of the first and second terms is found. The function in the parentheses in Equation (2) is called the evaluation function. The image processing device 34 can converge the solution through recursive iterative calculations and calculate the f that minimizes the evaluation function as the final solution f'.

[0201] The first term in the parentheses in Equation (2) represents an operation to calculate the sum of squares of the difference between the acquired data g and Hf, which is obtained by transforming f in the estimation process using matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation can obtain stable estimated data that suppresses the influence of noise in the observed data g. The sparsity of the object in the space of each regularization term varies depending on the texture of the object. A regularization term that makes the object texture sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0202] (Other 2) The timing chart shown in Fig. 8 may be as shown in Fig. 12. Fig. 12 shows a timing chart for performing reflectance hyperspectral imaging.

[0203] Emission start time a is the time when the high-brightness broadband light source 0111B starts emitting light. Emission end time b is the time when the high-brightness broadband light source 0111B stops emitting light. The emission period of the high-brightness broadband light source 0111B starts at emission start time a and ends at emission end time b. Exposure start time A is the time when the snapshot hyperspectral camera 0133B starts exposure. Exposure end time B is the time when the snapshot hyperspectral camera 0133B stops exposure. The exposure period of the snapshot hyperspectral camera 0133B starts at exposure start time A and ends at exposure end time B.

[0204] In Figures 8 and 12, among the exposure start time A, light emission start time a, light emission end time b, and exposure end time B, the exposure start time A is the earliest, the light emission start time a is the second earliest, the light emission end time b is the third earliest, and the exposure end time B is the fourth earliest.

[0205] However, the exposure start time A, the light emission start time a, the light emission end time b, and the exposure end time B may be set so that part of the exposure period and part of the light emission period overlap. In other words, the order of these four times may be either (a) or (b) below.

[0206] (a) Among the exposure start time A, light emission start time a, light emission end time b, and exposure end time B, the exposure start time A is the earliest, the light emission start time a is the second earliest, the exposure end time B is the third earliest, and the light emission end time b is the fourth earliest.

[0207] (b) Among the exposure start time A, light emission start time a, light emission end time b, and exposure end time B, light emission start time a is the earliest, exposure start time A is the second earliest, light emission end time b is the third earliest, and exposure end time B is the fourth earliest.

[0208] The control signal generator 0141 may cause the high-brightness broadband light source 0111B to start emitting light at an emission start time a. The control signal generator 0141 may cause the high-brightness broadband light source 0111B to end emitting light at an emission end time b. The control signal generator 0141 may cause the snapshot hyperspectral camera 0133B to start exposure at an exposure start time A. The control signal generator 0141 may cause the snapshot hyperspectral camera 0133B to end exposure at an exposure end time B.

[0209] The exposure end time B may be set to be earlier than the pupillary constriction reflex start time S, or the exposure end time B may be set to be the same as the pupillary constriction reflex start time S. This setting may be determined based on ΔT=((time t2)-(time t1)).

[0210] Time t1 is the time when light is emitted from the high-brightness broadband light source 0111B. Time t2 is the time when the pupillary constriction reflex caused by the light begins. ΔT may be determined by experiment. ΔT may be in the range of 0.1 seconds≦ΔT≦0.5 seconds.

[0211] (Other 3) The timing chart shown in Fig. 10 may be as shown in Fig. 13. Fig. 13 shows the timing when both reflectance hyperspectral imaging and autofluorescence imaging are performed.

[0212] Emission start time c is the time when the fluorescence excitation light source 0111C starts emitting light. Emission end time d is the time when the fluorescence excitation light source 0111C stops emitting light. The emission period of the fluorescence excitation light source 0111C starts at emission start time c and ends at emission end time d. Exposure start time C is the time when the compressed sensing hyperspectral camera 0433 starts exposure. Exposure end time D is the time when the compressed sensing hyperspectral camera 0433 stops exposure.

[0213] The exposure period of the compressed sensing hyperspectral camera 0433 starts at exposure start time C and ends at exposure end time D.

[0214] In FIGS. 10 and 13, the exposure start time C and the light emission start time c are simultaneous, and the light emission end time d and the exposure end time D are simultaneous.

[0215] However, the exposure start time C, exposure end time D, light emission start time c, and light emission end time d may be set so that part of the exposure period and part of the light emission period overlap. In other words, the order of these four times may be the following (c) or (d).

[0216] (c) Among the exposure start time C, exposure end time D, light emission start time c, and light emission end time d, exposure start time C is the earliest, light emission start time c is the second earliest, exposure end time D is the third earliest, and light emission end time d is the fourth earliest.

[0217] (d) Among the exposure start time C, exposure end time D, light emission start time c, and light emission end time d, light emission start time c is the earliest, exposure start time C is the second earliest, light emission end time d is the third earliest, and exposure end time D is the fourth earliest.

[0218] The control signal generator 0141 may cause the fluorescence excitation light source 0111C to start emitting light at an emission start time c. The control signal generator 0141 may cause the fluorescence excitation light source 0111C to end emitting light at an emission end time d. The control signal generator 0141 may cause the compressed sensing hyperspectral camera 0433 to start exposure at an exposure start time C. The control signal generator 0141 may cause the compressed sensing hyperspectral camera 0433 to end exposure at an exposure end time D.

[0219] The exposure end time D may be set to be earlier than the pupillary constriction reflex start time S, or the exposure end time D and the pupillary constriction reflex start time S may be set to be the same time. This setting may be determined based on ΔT=((time t2)-(time t1)).

[0220] Time t1 is the time when light is emitted from the high-brightness broadband light source 0111B. Time t2 is the time when the pupillary constriction reflex caused by the light begins. ΔT may be determined by experiment. ΔT may be in the range of 0.1 seconds≦ΔT≦0.5 seconds.

[0221] The imaging method and imaging system disclosed herein are effective for diagnosing and examining ophthalmic diseases.

[0222] 0110 Illumination unit 0111A Infrared light source 0111B High-intensity broadband emission light source 0111C Fluorescence excitation light source 0112A, 0112B, 0112C Collimator lens 0115, 0415 Optical path combining element 0116 Optical path adjustment element 0117 Condenser lens 0120 Common optical system 0121 Objective lens 0122 Beam splitter 0130 Imaging unit 0131 Focus lens 0132A, 0132B Imaging lens 0133A Infrared camera 0133B Snapshot hyperspectral camera 0136 Half mirror or dichroic mirror 0137 Optical path adjustment element 0140 Control unit 0141 Control signal generating device 0142 Display device 0210 Unit cell 0410 Illumination unit 0420 Common optical system 0430 Imaging unit 0433 Compressed hyperspectral camera 0434 Excitation light cut filter 0440 Control unit 0443 Memory

Claims

1. A method comprising: illuminating a subject's eye in a darkroom with a first light; and generating first image data indicative of the subject's fundus, the first image data including information in four or more wavelength bands, wherein the first image data is generated by having a camera take a snapshot of the subject's eye before pupil constriction caused by the illumination of the first light is completed.

2. The method of claim 1, wherein the first image data includes a compressed image in which information for the four or more wavelength bands is compressed.

3. The method of claim 1, wherein the first image data includes four or more spectral images that correspond one-to-one to the four or more wavelength bands.

4. The method of claim 1, further comprising illuminating the subject's eye with a second light comprising infrared light for adjusting the focus position of the camera.

5. The method of claim 1, wherein the first light includes excitation light, and the first image data is generated by causing the camera to capture fluorescence caused by the excitation light.

6. The method according to claim 5, further comprising: estimating a peak wavelength of the fluorescence from at least a portion of the fundus based on the first image data.

7. The method according to claim 6, wherein the peak wavelength is estimated by comparing information of two or more wavelength bands among the four or more wavelength bands included in the first image data.

8. The method of claim 1, further comprising: illuminating the subject's eye with a third light including excitation light; and generating second image data based on fluorescence caused by the third light, wherein the first image data and the second image data are generated by causing the camera to take a snapshot before pupil constriction of the eye caused by irradiation with the first light or the third light ends.

9. An imaging system comprising: a light source that irradiates a subject's eye in a darkroom with a first light; and a camera that generates first image data including information in four or more wavelength bands that indicates the fundus of the subject, wherein the camera generates the first image data by taking a snapshot of the subject's eye before pupil constriction of the eye caused by irradiation with the first light ends.

10. The imaging system according to claim 9, wherein the camera includes an optical element including four or more regions having mutually different transmission spectra.

11. The imaging system according to claim 10, wherein the transmission spectrum of each of the four or more regions has a plurality of maxima in the wavelength range of the imaging target, and the first image data includes a compressed image in which information of a plurality of wavelength bands is compressed.

12. The imaging system according to claim 10, wherein the peak wavelengths of the transmission spectra of the four or more regions are different from one another, and the first image data includes four or more spectral images that correspond one-to-one to the four or more wavelength bands.

13. The method of claim 1, wherein before the pupillary constriction ends is before the pupillary constriction begins.

14. The imaging system according to claim 9, wherein the period before the pupil contraction ends is the period before the pupil contraction starts.

Images