Scanning imaging device, information processing device, control method for scanning imaging device, control method for information processing device, information processing method, program, and recording medium

Abstract

To reduce errors in correction of motion artifacts.SOLUTION: An ophthalmologic examination device 1 is a scanning type imaging device that carries out imaging of a sample using optical scanning. The ophthalmologic examination device 1 redundantly collects data on the sample by optical scanning using an optical scanner 44, an OCT unit 100, and a control part 210, and acquires a dataset. An image data construction part 220 generates an image set on the basis of the acquired data set. An image selection part 225 selects a plurality of images from the generated image set. An image data processing part 231 executes relative position adjustment for the image set by applying a plurality of registrations with the plurality of selected images as references respectively, to the image set.SELECTED DRAWING: Figure 4B

Description

[Technical Field]

[0001] The present disclosure relates to a scanning imaging device, an information processing device, a control method for a scanning imaging device, a control method for an information processing device, an information processing method, a program, and a recording medium. [Background technology]

[0002] One imaging technique is scanning imaging, which involves sequentially irradiating a sample with a beam at multiple locations to collect data and then constructing an image of the sample from the collected data.

[0003] Scanning imaging techniques include, for example, spot-type scanning techniques that use a beam to project a spot-shaped image on a sample, line-type scanning techniques that use a beam to project a line-shaped image, and area-type scanning techniques that use a beam to project an area-shaped image. An example of a spot-type scanning technique is Fourier-domain optical coherence tomography (OCT), an example of a line-type scanning technique is a line scan camera, and an example of an area-type scanning technique is full-field OCT.

[0004] Optical Coherence Tomography (OCT) is an example of a scanning imaging modality that uses light. OCT is a technology that can image light-scattering media with a resolution of micrometers or less, and is used in medical imaging and non-destructive testing.

[0005] OCT is a technology based on low-coherence interferometry. In medical imaging, a probe light (measurement light) in the near-infrared region is used, taking into account its depth and invasiveness. However, scanning imaging using light (electromagnetic waves) in wavelength bands other than near-infrared, ultrasound, and radiation are also known.

[0006] Ophthalmology is one of the most advanced fields in the application of OCT, with products equipped with Fourier-domain OCT functionality now widespread in clinics. Ophthalmic imaging diagnostics not only provides two-dimensional imaging, but also three-dimensional imaging, structural analysis, and functional analysis, making it a widely used, powerful diagnostic tool. In addition to Fourier-domain OCT, scanning laser ophthalmoscopes (SLOs), line scan cameras, and full-field OCT are also used in ophthalmology.

[0007] There are various scanning modes used in OCT and SLO, but the so-called "Lissajous scan" for the purpose of correcting motion artifacts has been attracting attention in recent years (see, for example, Patent Documents 1 to 4 and Non-Patent Documents 1 and 2).

[0008] A Lissajous scan is a scan performed according to a two-dimensional pattern generated as a Lissajous curve, obtained by synthesizing two mutually orthogonal simple harmonic motions. In a Lissajous scan, the measurement light is scanned at high speed to trace multiple loops (multiple cycles that intersect with each other) of a certain size, so the difference in data acquisition time from one cycle can be virtually ignored. Furthermore, because the intersection areas of different cycles can be referenced to align the cycles, it is possible to correct motion artifacts caused by sample movement. Focusing on these characteristics of the Lissajous scan, the field of ophthalmology is attempting to address motion artifacts caused by eye movement.

[0009] In the technique described in Non-Patent Document 1, a data set acquired by a Lissajous scan is divided into multiple sub-volumes that do not involve relatively large motion, and a front projection image (en face projection) of each sub-volume is constructed. Such front projection images are called strips. Images with motion artifacts corrected can be obtained by performing registration between these strips. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-17915 [Patent Document 2] Japanese Patent Application Publication No. 2018-68578 [Patent Document 3] Japanese Patent Application Laid-Open No. 2018-140004 [Patent Document 4] Japanese Patent Application Laid-Open No. 2018-140049 [Non-patent literature]

[0011] [Non-Patent Document 1] Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Three-dimensional eye motion correction by Lissajous scan optical coherence tomography”, Biomedical Optics EXPRESS, Vol. 8, No. 3, 1 Mar 2017, PP. 1783-1802 [Non-patent document 2] Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Eye-motion-corrected optical coherence tomography angiography using Lissajous scanning”, Biomedical Optics EXPRESS, Vol. 9, No. 3, 1 Mar 2018, PP. 1111-1129 Summary of the Invention [Problem to be solved by the invention]

[0012] An object of the present invention is to reduce errors in motion artifact correction. [Means for solving the problem]

[0013] An exemplary embodiment of a scanning imaging device is a scanning imaging device that images a sample using optical scanning, and includes a dataset acquisition unit that redundantly collects data of the sample using the optical scanning to acquire a dataset, an image set generation unit that generates an image set based on the dataset, an image selection unit that selects multiple images from the image set, and an image position adjustment unit that adjusts the relative position of the image set by applying multiple registrations to the image set based on each of the multiple images.

[0014] The image position adjustment unit may be configured to generate multiple pieces of relative position information for the image set by applying the multiple registrations to the image set, and to perform the relative position adjustment based on the multiple pieces of relative position information. The image position adjustment unit may be configured to generate the relative position information for the image set by applying recursive registration to the image set using one of the multiple images as an initial reference image for each of the multiple registrations. The image position adjustment unit may be configured to determine an adjustment amount for the relative position adjustment based on the multiple pieces of relative position information. The image selection unit may be configured to select the multiple images from the image set based on a time axis of the optical scanning. The image selection unit may be configured to select the multiple images from the image set by dividing the image set into multiple subsets corresponding to multiple sections of the time axis and selecting an image from each of the multiple subsets. The image selection unit may be configured to select the multiple images from the image set based on an application area of ​​the optical scanning. The image selector may be configured to select the images from the image set by dividing the image set into a plurality of subsets corresponding to a plurality of sub-areas of the application area and selecting an image from each of the plurality of subsets. The image selector may be configured to select an image from each of the plurality of subsets based on image size. The image selector may be configured to select an image from each of the plurality of subsets with a largest size in the subset. The image selector may be configured to order the images included in each of the plurality of subsets based on image size, and to select an image from each of the plurality of subsets based on the order assigned to the images in the subset by the ordering.The image selector may be configured to determine whether registration is successful or unsuccessful based on a first image selected from a first subset of the plurality of subsets, and if the registration is determined to be unsuccessful, select a second image from the first subset based on the order assigned to the images in the first subset. The image selector may be configured to order images included in the image set based on image size and select the plurality of images from the image set based on the order assigned to the images in the ordering. The image selector may be configured to determine whether registration is successful or unsuccessful based on one of the plurality of images selected from the image set, and if the registration is determined to be unsuccessful, select a new image from the image set that is different from any of the plurality of images based on the order assigned to the images in the image set. The scanning imaging apparatus of the embodiment may further include a motion information generator that generates motion information representing motion of the sample when the optical scanning is applied to the sample, and the image selector may be configured to select the plurality of images from the image set based on the motion information. The image selector may be configured to calculate a value of a predetermined motion parameter from the motion information, evaluate the value, and select the plurality of images from the image set based on a result of the evaluation. The motion parameter may include at least one of a magnitude and a frequency of motion, and the image selector may be configured to compare the value of the motion parameter with a predetermined threshold and select the plurality of images from a subset obtained by excluding, from the image set, images based on data collected during a period corresponding to the value equal to or greater than the threshold. The dataset acquirer may include a deflector capable of deflecting light for the optical scanning in a first direction and a second direction different from each other, and may be configured to repeatedly change the deflection direction in the first direction in a first cycle and repeatedly change the deflection direction in the second direction in a second cycle different from the first cycle.

[0015] An information processing device according to an exemplary embodiment includes a dataset receiving unit that receives a dataset acquired by redundantly collecting data from a sample by optical scanning, an image set generating unit that generates an image set based on the dataset, an image selecting unit that selects a plurality of images from the image set, and an image position adjusting unit that adjusts the relative position of the image set by applying a plurality of registrations to the image set based on each of the plurality of images.

[0016] An information processing device according to an exemplary embodiment includes an image set receiving unit that receives an image set generated based on a data set acquired by redundantly collecting data from a sample by optical scanning, an image selecting unit that selects a plurality of images from the image set, and an image position adjusting unit that adjusts the relative position of the image set by applying a plurality of registrations to the image set based on each of the plurality of images.

[0017] A control method for a scanning imaging device according to an exemplary embodiment is a method for controlling a scanning imaging device including a processor and a scanner that performs optical scanning, the method comprising: ,sa The method further comprises redundantly collecting sample data to obtain a dataset, causing the processor to generate an image set based on the dataset, causing the processor to select a plurality of images from the image set, and causing the processor to perform relative position adjustment of the image set by applying a plurality of registrations to the image set, each of which is based on the plurality of images.

[0018] An exemplary embodiment of a control method for an information processing device is a method for controlling an information processing device including a processor, and includes causing the processor to accept a dataset acquired by redundantly collecting data from a sample by optical scanning, causing the processor to generate an image set based on the dataset, causing the processor to select multiple images from the image set, and performing relative position adjustment of the image set by applying multiple registrations to the image set based on each of the multiple images.

[0019] An exemplary embodiment of a control method for an information processing device is a method for controlling an information processing device including a processor, which causes the processor to accept an image set generated based on a data set acquired by redundantly collecting data from a sample by optical scanning, causes the processor to select multiple images from the image set, and performs relative position adjustment of the image set by applying multiple registrations to the image set based on each of the multiple images.

[0020] An exemplary embodiment of an information processing method is an information processing method for processing a dataset acquired by redundantly collecting data of a sample by optical scanning, generating an image set based on the dataset, selecting multiple images from the image set, and adjusting the relative position of the image set by applying multiple registrations to the image set based on each of the multiple images.

[0021] The program of the exemplary embodiment is a program that causes a computer to execute any of the methods of the present embodiment.

[0022] The recording medium of the exemplary embodiment is a computer-readable non-transitory recording medium on which any of the programs of the present embodiment is recorded. [Effects of the Invention]

[0023] According to an exemplary embodiment, errors in motion artifact correction can be reduced. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 2] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 3] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 4A] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 4B] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 4C] 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 5] 1 is a schematic diagram showing an example of a Lissajous scan pattern performed by a scanning imaging device (ophthalmic examination device) according to an exemplary aspect of the embodiment. [Figure 6] 1 is a schematic diagram showing an example of an initial reference image selection process performed by a scanning imaging device (ophthalmic examination device) according to an exemplary aspect of an embodiment. [Figure 7] 1 is a schematic diagram showing an example of an initial reference image selection process performed by a scanning imaging device (ophthalmic examination device) according to an exemplary aspect of an embodiment. [Figure 8A] 1 is a schematic diagram showing an example of an initial reference image selection process performed by a scanning imaging device (ophthalmic examination device) according to an exemplary aspect of an embodiment. [Figure 8B] 1 is a schematic diagram showing an example of an initial reference image selection process performed by a scanning imaging device (ophthalmic examination device) according to an exemplary aspect of an embodiment. [Figure 9A] 10A and 10B are schematic diagrams illustrating an example of an adjustment amount calculation process executed by a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 9B] 10A and 10B are schematic diagrams illustrating an example of an adjustment amount calculation process executed by a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 9C] 10A and 10B are schematic diagrams illustrating an example of an adjustment amount calculation process executed by a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 9D] 10A and 10B are schematic diagrams illustrating an example of an adjustment amount calculation process executed by a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 10] 10A and 10B are schematic diagrams illustrating an example of an adjustment amount calculation process executed by a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 11] 1 is a flowchart illustrating an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 12] 1 is a flowchart illustrating an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 13A] 1 is a flowchart illustrating an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 13B] 1 is a flowchart illustrating an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 14A] 1A to 1C are diagrams for explaining an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 14B] 1A to 1C are diagrams for explaining an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 14C] 1A to 1C are diagrams for explaining an example of the operation of a scanning imaging apparatus (ophthalmic examination apparatus) according to an exemplary aspect of an embodiment. [Figure 15]1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary aspect of an embodiment. [Figure 16] 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary aspect of an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] Some exemplary aspects of the embodiments will be described with reference to the drawings. The exemplary aspects described in the present disclosure relate to a scanning imaging device, an information processing device, a control method for a scanning imaging device, a control method for an information processing device, an information processing method, a program, and a recording medium, but the embodiments are not limited to these aspects.

[0026] Although some exemplary aspects described in the present disclosure primarily provide applications in the ophthalmology field, embodiments are not limited to these aspects. For example, some exemplary embodiments may provide applications in any field in which scanning imaging is or may be used, such as applications in medical fields other than ophthalmology, medical methods such as diagnostic methods, and fields other than the medical field (biology, non-destructive testing, etc.).

[0027] The matters disclosed in the documents cited in this disclosure and matters relating to any other known technology may be combined with the embodiments. In this disclosure, unless otherwise specified, no distinction is made between a "site" of a subject (examined eye), "image data" corresponding to that site, and an "image" which is a visual representation of that image data.

[0028] A scanning imaging apparatus according to an exemplary embodiment described below is an apparatus used in ophthalmic examinations (referred to as an ophthalmic examination apparatus), and is configured to be capable of measuring and imaging the fundus of a living eye using Fourier-domain OCT (particularly, swept-source OCT). The type of OCT that can be employed in the embodiments is not limited to swept-source OCT, and may be, for example, spectral-domain OCT or time-domain OCT. Furthermore, although the scanning imaging apparatus according to the exemplary embodiment described below uses a spot-type scanning technique, it will be understood by those skilled in the art that any aspect of the present disclosure can be applied to a scanning imaging apparatus that uses other types of scanning techniques (e.g., a line-type scanning technique or an area-type scanning technique).

[0029] A scanning imaging device according to some exemplary embodiments may be able to use a scanning imaging modality other than OCT. Some exemplary embodiments may employ any optical scanning imaging modality, such as SLO. Furthermore, the scanning imaging modality applicable to some exemplary embodiments may not be an optical scanning imaging modality, but may be, for example, a scanning imaging modality that uses electromagnetic waves other than light, a scanning imaging modality that uses ultrasound, or the like.

[0030] In some exemplary embodiments, the ophthalmic examination apparatus may be capable of processing data acquired by other imaging modalities in addition to processing data acquired by OCT scans and / or SLO. The "other imaging modalities" may be any ophthalmic imaging modality, such as a fundus camera, a slit lamp microscope, or an ophthalmic surgical microscope. The ophthalmic examination apparatus according to some exemplary embodiments may have the functionality of imaging modalities other than OCT and SLO. Furthermore, the ophthalmic examination apparatus according to some exemplary embodiments may have a stereo alignment function using two or more anterior segment cameras, as disclosed in Japanese Patent Application Laid-Open No. 2013-248376. Stereo alignment is a function for identifying the three-dimensional position of the subject's eye by analyzing two or more anterior segment images obtained by two or more anterior segment cameras, and aligning the optical system.

[0031] The object (sample) to which scanning imaging is applied may be any object. In the ophthalmology applications described in the following exemplary embodiments, scanning imaging is mainly applied to the fundus, but the object to which scanning imaging is applied is not limited to the fundus and may be any part of the eye, such as the anterior segment or the vitreous body. The configurations and functions of some exemplary embodiments may be applied to measurement and imaging of biological parts (tissues) other than the eye, and may also be applied to measurement and imaging of non-biological objects (e.g., objects that move) or parts thereof. That is, the industrial application fields of some exemplary embodiments are not limited to ophthalmology-related fields, but may also include medical, veterinary, and biological fields, and may more generally include fields related to any object (sample) that moves locally and / or globally. The industrial application fields of some exemplary embodiments may also include fields related to objects that do not move. For example, the object may be an object that moves or deforms due to influences from surrounding objects or the environment. It should be noted that, since one of the purposes of the scanning imaging of the present disclosure is to improve motion artifact correction, a typical object in the present disclosure may be an object with active motion or an object with passive motion, or may be an object including an object with active motion and / or an object with passive motion.

[0032] Any aspects of the embodiments described below are merely illustrative and are not intended to limit the invention.

[0033] As described above, one objective of the scanning imaging disclosed herein is to improve motion artifact correction. In conventional motion artifact correction for Lissajous scanning, data (volume) collected by the Lissajous scan is divided into multiple segments (multiple subvolumes) of a size that allows the influence of object motion to be ignored. One of these segments is selected as an initial reference image (sometimes referred to as an initial reference image, initial reference image, initial reference strip, initial reference segment, etc.). This single initial reference image is used as the starting point (the reference image in the first processing iteration), and recursive estimation of the misalignment between segments (registration and merging processes) is performed. In this recursive process (iterative process), other segments are sequentially registered and merged with the selected single initial reference image, thereby expanding the range represented by the successively formed reference images. With this method, it is conceivable that the selection of the initial reference image may adversely affect the quality of the final image. For example, according to the inventors' investigations, if a sufficiently appropriate initial selection image is not selected, segments with large registration errors may be merged as they are, resulting in the final image being represented as clearly discontinuous structures.

[0034] The technology according to the present disclosure has been discovered by the inventors to address these problems, and an overview thereof will be first described here. Note that a more detailed and accurate description of the technology according to the present disclosure will be provided in the non-limiting exemplary embodiments described below.

[0035] In general, the technology disclosed herein is configured to divide data collected from an object by scanning into multiple segments, obtain multiple initial reference images from the multiple segments, and perform recursive inter-segment misalignment estimation starting from each initial reference image. In other words, the technology disclosed herein is configured to divide the collected data into multiple segments, select multiple images from the multiple segments, and perform multiple registrations (and merging processes) based on the multiple images, respectively, to adjust the relative positions of the multiple segments. Although motion artifacts may still remain even when using the technology disclosed herein, these residual errors can be corrected by detailed motion artifact correction that can be performed in a subsequent stage.

[0036] In some exemplary aspects, a plausible displacement amount can be determined from multiple displacement estimates obtained by multiple registrations. Furthermore, a method for selecting multiple initial reference images may be arbitrary, and the present disclosure provides several examples. For example, the present disclosure describes, as non-limiting initial reference image selection methods, a method for selecting an initial reference image from each of multiple sections obtained by dividing collected data in time, or a method for selecting an initial reference image from each of multiple regions obtained by dividing collected data in space.

[0037] The technology disclosed herein can reduce errors in motion artifact correction compared to conventional technologies, thereby improving imaging accuracy (measurement accuracy). When applied to the field of ophthalmology, it can improve OCT image measurement accuracy and diagnostic accuracy in standard ophthalmic image diagnosis. Furthermore, the technology disclosed herein can be implemented without adding special hardware to current ophthalmic OCT devices and can be put into practical use relatively inexpensively, making it possible to envision its use in ophthalmic examinations and other applications. Furthermore, since it can achieve motion artifact correction with higher accuracy than conventional OCT devices, it is possible to stably perform high-precision image measurement. This makes it possible, for example, to track minute changes in lesions over time, and even to track minute changes in lesions over the medium to long term. Those skilled in the art will readily understand that similar effects and advantages can be achieved in applications other than ophthalmology.

[0038] This disclosure describes some exemplary aspects of embodiments of a scanning imaging apparatus. The scanning imaging apparatus of this embodiment is an apparatus that performs imaging (image measurement, optical measurement) of an object (sample) using optical scanning.

[0039] The scanning imaging apparatus of this embodiment is configured to acquire a data set by redundantly collecting data of a sample through optical scanning. This redundant data collection is a data collection mode in which data is collected two or more times from a specific location on the sample. In this embodiment, data is collected two or more times from each of multiple locations on the sample. The Lissajous scan described in Patent Documents 1 to 4 and Non-Patent Documents 1 and 2 provides an example of redundant data collection. In a Lissajous scan, the location from which data is collected redundantly corresponds to the intersection position between one cycle and another cycle. Motion artifact correction using a Lissajous scan involves registering data redundantly acquired from a certain intersection position to determine the amount of shift between cycles that intersect at that intersection position.

[0040] Furthermore, the scanning imaging apparatus of this embodiment is configured to generate an image set based on a data set acquired by redundant data acquisition, where the image set includes multiple images with positional redundancy, and may include, for example, multiple strips or multiple sub-volumes, as described below.

[0041] Additionally, the scanning imaging apparatus of this embodiment is configured to select multiple images from the generated image set. Each of the multiple images selected here corresponds to the initial reference image described above. The number of multiple initial reference images selected from the image set may be arbitrary. For example, only some of the multiple images included in the image set may be designated as initial reference images, or all of the multiple images included in the image set may be designated as initial reference images. Each initial reference image is used as a starting point for recursive registration (a process described below that combines recursively performed registration and recursively performed merging) performed in subsequent processing. That is, each initial reference image is used as a reference image in the first registration of the recursive registration.

[0042] Furthermore, the scanning imaging device of this embodiment is configured to perform relative position adjustment of the image set by applying multiple registrations to the image set, each based on a selected number of images (a number of initial reference images).

[0043] <Configuration of ophthalmic examination device> The ophthalmic examination apparatus 1 shown in FIG. 1 is an exemplary embodiment of a scanning imaging apparatus according to the present invention. The ophthalmic examination apparatus 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic and control unit 200. The fundus camera unit 2 includes a group of elements (optical elements, mechanisms, etc.) for capturing an image of the subject's eye E from the front. The OCT unit 100 includes some of the group of elements (optical elements, mechanisms, etc.) for performing an OCT scan on the subject's eye E. At least some of the other elements for the OCT scan are provided in the fundus camera unit 2. The arithmetic and control unit 200 includes one or more processors that perform various calculations and controls. In addition to these, the ophthalmic examination apparatus 1 may also include elements for supporting the subject's face and elements for switching the region to which the OCT scan is applied. Examples of the former elements include a chin rest and a forehead rest. An example of the latter element is a lens unit used to switch the region to which the OCT scan is applied from the fundus to the anterior segment.

[0044] The functionality of some elements described in this disclosure is implemented using circuitry or processing circuitry. Circuitry or processing circuitry includes any of a general-purpose processor, a special-purpose processor, an integrated circuit, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a programmable logic device (e.g., a SPLD (Simple Programmable Logic Device), a CPLD (Complex Programmable Logic Device), an FPGA (Field Programmable Gate Array), conventional circuitry, and any combination thereof, configured and / or programmed to perform the disclosed functions. A processor is considered to be processing circuitry or circuitry that includes transistors and / or other circuitry. In this disclosure, circuitry, unit, means, or similar terms is hardware that performs the disclosed functions or hardware that is programmed to perform the disclosed functions. The hardware may be any hardware described in this disclosure or known hardware that is programmed and / or configured to perform the functions described in this disclosure. In the case of a processor, where the hardware can be considered to be a type of circuitry, the circuitry, unit, means, or similar terms is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0045] <Fundus camera unit 2> The fundus camera unit 2 is provided with an optical system for photographing the fundus Ef of the subject's eye E. The acquired image of the fundus Ef (called a fundus image, fundus photograph, etc.) is a front image such as an observed image or a photographed image. The observed image is obtained by, for example, video shooting using near-infrared light, and is used for alignment, focusing, tracking, etc. The photographed image is a still image obtained using, for example, flash light in the visible or infrared range.

[0046] The fundus camera unit 2 includes an illumination optical system 10 and an imaging optical system 30. The illumination optical system 10 irradiates illumination light onto the subject's eye E. The imaging optical system 30 detects return light of the illumination light from the subject's eye E. The measurement light from the OCT unit 100 is guided to the subject's eye E through an optical path within the fundus camera unit 2, and the return light is guided to the OCT unit 100 through the same optical path.

[0047] Light (observation illumination light) output from an observation light source 11 of an illumination optical system 10 is reflected by a concave mirror 12, passes through a condenser lens 13, and is transmitted through a visible light cut filter 14 to become near-infrared light. The observation illumination light is then focused near an imaging light source 15, reflected by a mirror 16, and passes through a relay lens system 17, a relay lens 18, an aperture 19, and a relay lens system 20. The observation illumination light is then reflected by the peripheral portion (the area surrounding the hole) of a perforated mirror 21, passes through a dichroic mirror 46, and is refracted by an objective lens 22 to illuminate the subject's eye E (fundus oculi Ef). Return light of the observation illumination light from the subject's eye E is refracted by the objective lens 22, passes through the dichroic mirror 46, passes through a hole formed in the central region of the perforated mirror 21, passes through a dichroic mirror 55, passes through an imaging focusing lens 31, and is reflected by a mirror 32. Furthermore, this returned light passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged on the light receiving surface of the image sensor 35 by the imaging lens 34. The image sensor 35 detects the returned light at a predetermined frame rate. The focus (focal position) of the photographing optical system 30 is adjusted to match the fundus Ef or the anterior segment of the eye.

[0048] Light (photography illumination light) output from the photography light source 15 is irradiated onto the fundus oculi Ef through the same path as the observation illumination light. Return light of the photography illumination light from the subject's eye E is guided to the dichroic mirror 33 through the same path as the return light of the observation illumination light, passes through the dichroic mirror 33, is reflected by a mirror 36, and is imaged by an imaging lens 37 on the light-receiving surface of an image sensor 38.

[0049] The liquid crystal display (LCD) 39 displays a fixation target (fixation target image). A portion of the light beam output from the LCD 39 is reflected by the half mirror 33A, reflected by the mirror 32, passes through the photographing focusing lens 31 and the dichroic mirror 55, and passes through the hole in the aperture mirror 21. The light beam that passes through the hole in the aperture mirror 21 passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the fundus Ef. By changing the display position of the fixation target image on the LCD 39, the direction in which the gaze of the subject's eye E is guided (fixation direction, fixation position) can be changed. Instead of a display device such as an LCD, for example, a light-emitting element array, or a combination of a light-emitting element and a mechanism for moving it, may be used.

[0050] The alignment optical system 50 generates an alignment index used to align the optical system with the subject's eye E. Alignment light output from a light-emitting diode (LED) 51 passes through an aperture 52, an aperture 53, and a relay lens 54, is reflected by a dichroic mirror 55, passes through the hole in the aperture mirror 21, transmits through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. Return light of the alignment light from the subject's eye E (such as corneal reflection light) is guided to the image sensor 35 via the same path as the return light of the observation illumination light. Manual alignment or automatic alignment can be performed based on the received light image (alignment index image).

[0051] As in the conventional example, the alignment target image in this example consists of two bright spot images whose positions change depending on the alignment state. When the relative position between the subject's eye E and the optical system changes in the x and y directions (directions expressed by the x and y coordinates), the two bright spot images displace together in the x and y directions. When the relative position between the subject's eye E and the optical system changes in the z direction (direction expressed by the z coordinate), the relative position (distance) between the two bright spot images changes. When the distance between the subject's eye E and the optical system in the z direction matches the predetermined working distance, the two bright spot images overlap. When the position of the subject's eye E matches the position of the optical system in the x and y directions, the two bright spot images are presented within or near a specified alignment target. When the distance between the subject's eye E and the optical system in the z direction matches the working distance and the position of the subject's eye E matches the position of the optical system in the x and y directions, the two bright spot images overlap and are presented within the alignment target.

[0052] In auto-alignment, the data processing unit 230 detects the positions of the two bright spot images, and the main control unit 211 controls the moving mechanism 150 (described later) based on the positional relationship between the two bright spot images and the alignment target. In manual alignment, the main control unit 211 displays the two bright spot images together with the observed image of the subject's eye E on the display unit 241, and the user operates the moving mechanism 150 using the operation unit 242 while referring to the two displayed bright spot images.

[0053] The focusing optical system 60 generates a split index used for focus adjustment of the subject's eye E. The focusing optical system 60 moves along the optical path (illumination optical path) of the illumination optical system 10 in conjunction with movement of the photographing focusing lens 31 along the optical path (photographing optical path) of the photographing optical system 30. The reflecting rod 67 is inserted into and removed from the illumination optical path. When performing focus adjustment, the reflecting surface of the reflecting rod 67 is tilted and positioned in the illumination optical path. Focusing light output from the LED 61 passes through the relay lens 62, is split into two beams by the split index plate 63, passes through the two-hole diaphragm 64, is reflected by the mirror 65, and is first imaged and reflected on the reflecting surface of the reflecting rod 67 by the condenser lens 66. The focusing light then passes through the relay lens 20, is reflected by the aperture mirror 21, passes through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. The return light (fundus reflected light, etc.) of the focusing light from the subject's eye E passes through the same path as the return light of the alignment light and is guided to the image sensor 35. Manual focusing or autofocusing can be performed based on the received light image (split target image).

[0054] Diopter correction lenses 70 and 71 can be selectively inserted into the photographing optical path between the aperture mirror 21 and the dichroic mirror 55. The diopter correction lens 70 is a plus lens (convex lens) for correcting severe hyperopia. The diopter correction lens 71 is a minus lens (concave lens) for correcting severe myopia.

[0055] The dichroic mirror 46 combines the optical path for fundus imaging and the optical path for OCT (measurement arm). The dichroic mirror 46 reflects light in the wavelength band used for OCT and transmits light for fundus imaging. The measurement arm is provided with, in order from the OCT unit 100 side, a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45.

[0056] The retroreflector 41 is movable in the direction of the arrow shown in Figure 1, thereby changing the length of the measurement arm. Changing the length of the measurement arm is used, for example, to correct the optical path length according to the axial length of the eye, adjust the interference state, and so on.

[0057] The dispersion compensation member 42, together with a dispersion compensation member 113 (described later) disposed in the reference arm, acts to match the dispersion characteristics of the measurement light LS with the dispersion characteristics of the reference light LR.

[0058] The OCT focusing lens 43 is moved along the measurement arm to adjust the focus of the measurement arm. The movement of the imaging focusing lens 31, the movement of the focus optical system 60, and the movement of the OCT focusing lens 43 can be controlled in a coordinated manner.

[0059] The optical scanner 44 is disposed at a position that is substantially optically conjugate with the pupil of the subject's eye E. The optical scanner 44 deflects the measurement light LS guided by the measurement arm. The optical scanner 44 is, for example, a galvanometer scanner capable of two-dimensional scanning, including a galvanometer mirror for scanning in the x direction and a galvanometer mirror for scanning in the y direction.

[0060] <OCTユニット100> As shown in FIG. 2, the OCT unit 100 is provided with an optical system for applying swept-source OCT. This optical system includes an interference optical system. This interference optical system splits light from a wavelength-tunable light source (swept-wavelength light source) into measurement light and reference light, and generates interference light by superimposing the return light of the measurement light guided to the subject's eye E by the measurement arm on the reference light guided by the reference arm, and detects this interference light. Data (detection signal) obtained by the interference optical system is a signal representing the spectrum of the interference light, and is sent to the arithmetic and control unit 200.

[0061] The light source unit 101 includes, for example, a near-infrared tunable laser that changes the output wavelength at high speed at least in the near-infrared wavelength band. Light L0 output from the light source unit 101 is guided by an optical fiber 102 to a polarization controller 103, where its polarization state is adjusted. The light L0 is further guided by an optical fiber 104 to a fiber coupler 105, where it is split into measurement light LS and reference light LR. The optical path of the measurement light LS is called a measurement arm, and the optical path of the reference light LR is called a reference arm.

[0062] The reference light LR is guided by an optical fiber 110 to a collimator 111, where it is converted into a parallel beam, and then guided to a retroreflector 114 via an optical path length correction element 112 and a dispersion compensation element 113. The optical path length correction element 112 acts to match the optical path length of the reference light LR with that of the measurement light LS. The dispersion compensation element 113, together with a dispersion compensation element 42 arranged in the measurement arm, acts to match the dispersion characteristics between the reference light LR and the measurement light LS. The retroreflector 114 is movable along the optical path of the reference light LR incident thereon, thereby changing the length of the reference arm. Changing the reference arm length is used, for example, to correct the optical path length according to the axial length of the eye, adjust the interference state, and so on.

[0063] The reference light LR that has passed through the retroreflector 114 passes through the dispersion compensation member 113 and the optical path length correction member 112, is converted from a parallel beam into a convergent beam by the collimator 116, and enters an optical fiber 117. The reference light LR that has entered the optical fiber 117 is guided to a polarization controller 118 where its polarization state is adjusted, is guided through an optical fiber 119 to an attenuator 120 where its light amount is adjusted, and is guided through an optical fiber 121 to a fiber coupler 122.

[0064] On the other hand, the measurement light LS generated by the fiber coupler 105 is guided by the optical fiber 127 and converted into a parallel beam by the collimator lens unit 40, passes through the retroreflector 41, the dispersion compensation member 42, the OCT focusing lens 43, the optical scanner 44, and the relay lens 45, is reflected by the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. The measurement light LS is scattered and reflected at various depth positions in the subject's eye E. The returning light of the measurement light LS from the subject's eye E travels the same path as the outward path in the opposite direction and is guided to the fiber coupler 105, and reaches the fiber coupler 122 via the optical fiber 128.

[0065] The fiber coupler 122 generates interference light by superimposing the measurement light LS incident via the optical fiber 128 and the reference light LR incident via the optical fiber 121. The fiber coupler 122 splits the generated interference light at a predetermined splitting ratio (for example, 1:1) to generate a pair of interference lights LC. The pair of interference lights LC are guided to the detector 125 via optical fibers 123 and 124, respectively.

[0066] The detector 125 includes, for example, a balanced photodiode. The balanced photodiode has a pair of photodetectors that respectively detect a pair of interference light LC, and outputs the difference between the pair of detection results obtained by these. The detector 125 sends this output (detection signal) to a data acquisition system (DAS) 130.

[0067] A clock KC is supplied to the data collection system 130 from the light source unit 101. The clock KC is generated in the light source unit 101 in synchronization with the output timing of each wavelength swept within a predetermined wavelength range by the wavelength-tunable light source. The light source unit 101, for example, branches the light L0 of each output wavelength to generate two branched lights, optically delays one of the branched lights, combines the branched lights, detects the resulting combined light, and generates the clock KC based on the detection result. The data collection system 130 samples the detection signal input from the detector 125 based on the clock KC. The data collection system 130 sends the sampling result to the arithmetic and control unit 200.

[0068] In this example, both an element for changing the measurement arm length (e.g., retroreflector 41) and an element for changing the reference arm length (e.g., retroreflector 114 or reference mirror) are provided, but only one of these elements may be provided. Furthermore, the element for changing the difference (optical path length difference) between the measurement arm length and the reference arm length is not limited to these exemplary elements, and may be any element (optical member, mechanism, etc.) that can realize the function of changing the arm length.

[0069] <Control system / processing system> 3, 4A, 4B, and 4C show configuration examples of the control system and processing system of the ophthalmic examination apparatus 1. The control unit 210, image data creation unit 220, and data processing unit 230 are provided in, for example, the arithmetic control unit 200. The ophthalmic examination apparatus 1 may include a communication device for performing data communication with an external device. The ophthalmic examination apparatus 1 may also include a drive device (reader / writer) for reading data from a recording medium and writing data to the recording medium.

[0070] <Control unit 210> The control unit 210 executes various types of control. The control unit 210 includes a main control unit 211 and a storage unit 212. In this embodiment, the main control unit 211 includes a scanning control unit 2111, and the storage unit 212 stores a scanning protocol 2121, as shown in FIG. 4A.

[0071] <Main control unit 211> The main control unit 211 includes a processor and controls each element of the ophthalmic examination apparatus 1 (including the elements shown in FIGS. 1 to 4X). The main control unit 211 is realized by cooperation between hardware including the processor and control software. The scan control unit 2111 controls OCT scanning of a scan area of ​​a predetermined shape and size.

[0072] The imaging focusing driver 31A moves the imaging focusing lens 31 arranged in the imaging optical path and the focus optical system 60 arranged in the illumination optical path under the control of the main controller 211. The retroreflector (RR) driver 41A moves the retroreflector 41 provided in the measurement arm under the control of the main controller 211. The OCT focusing driver 43A moves the OCT focusing lens 43 arranged in the measurement arm under the control of the main controller 211. The retroreflector (RR) driver 114A moves the retroreflector 114 arranged in the reference arm under the control of the main controller 211. Each driver includes an actuator such as a pulse motor that operates under the control of the main controller 211. The optical scanner 44 operates under the control of the main controller 211 (scanning controller 2111).

[0073] The movement mechanism 150 is typically configured to move the fundus camera unit 2 three-dimensionally. Such a movement mechanism 150 includes, for example, an x-stage configured to be movable in ±x directions (left and right directions), an x-movement mechanism configured to move the x-stage, a y-stage configured to be movable in ±y directions (up and down directions), a y-movement mechanism configured to move the y-stage, a z-stage configured to be movable in ±z directions (depth direction), and a z-movement mechanism configured to move the z-stage. Each of the x-movement mechanism, y-movement mechanism, and z-movement mechanism includes an actuator such as a pulse motor that operates under the control of the main controller 211.

[0074] <Storage section 212> The storage unit 212 stores various types of data. The data stored in the storage unit 212 includes OCT images, fundus images, subject eye information, and control information. The subject eye information includes subject information such as patient ID and name, left / right eye identification information, and electronic medical record information. The control information is information related to specific controls. The control information in this embodiment includes a scanning protocol 2121.

[0075] The scanning protocol 2121 is an agreement regarding the control content for an OCT scan of a scanning area of ​​a predetermined shape and a predetermined size, and includes a set of various control parameters (scanning control parameters). The scanning protocol 2121 includes a protocol for each scanning mode. The scanning protocol 2121 of this embodiment includes at least a protocol for Lissajous scanning, and may further include, for example, a protocol for B scanning (line scanning), a protocol for cross scanning, a protocol for radial scanning, a protocol for raster scanning, etc.

[0076] The scanning control parameters of this embodiment include at least a parameter indicating the content of control over the optical scanner 44. These parameters include, for example, a parameter indicating the scan pattern, a parameter indicating the scan speed, and a parameter indicating the scan interval. The scan pattern indicates the shape of the scan path, and examples thereof include a Lissajous pattern, a line pattern, a cross pattern, a radial pattern, and a raster pattern. The scan speed is defined, for example, as the repetition rate of A-scans. The scan interval is defined, for example, as the interval between adjacent A-scans, that is, the array interval of scan points.

[0077] As with the conventional technologies disclosed in Patent Documents 1 to 4 and Non-Patent Documents 1 and 2, the "Lissajous scan" of this embodiment may be not only a "narrowly defined" Lissajous scan whose path is a pattern (Lissajous pattern, Lissajous figure, Lissajous curve, Lissajous function, Bowditch curve) drawn by the locus of points obtained by an ordered pair of two mutually perpendicular simple harmonic motions, but also a "broadly defined" Lissajous scan that follows a predetermined two-dimensional pattern including a series of cycles.

[0078] The optical scanner 44 of this embodiment includes, for example, a first galvanometer mirror that deflects the measurement light LS in the x direction and a second galvanometer mirror that deflects the measurement light LS in the y direction. Lissajous scanning is achieved by controlling the first galvanometer mirror so that the deflection direction along the x direction repeatedly changes in a first period, while controlling the second galvanometer mirror so that the deflection direction along the y direction repeatedly changes in a second period. Here, the first period and the second period are different from each other.

[0079] For example, the Lissajous scan of this embodiment may not only be a scan of a narrowly defined Lissajous pattern obtained from a combination of two sine waves, but also a scan of a pattern obtained by adding a specific term (e.g., an odd-order polynomial) to a sine wave, or a scan of a pattern based on a triangular wave.

[0080] A "cycle" generally refers to an object consisting of a plurality of sampling points having a certain length. The cycle in this embodiment may be, for example, a closed curve or an almost closed curve (a substantially closed curve, an approximately closed curve). In other words, the start point and end point of the cycle in this embodiment may coincide or nearly coincide.

[0081] Typically, the scanning protocol 2121 is set based on the Lissajous function. As shown in equations (9) and (10) in Non-Patent Document 1, the Lissajous function is expressed, for example, by the following parametric equation system: x(t i ) = A·cos(2π·(f A / n)·t i ), y(t i ) = A·cos(2π·(f A (n-2) / n 2 )·t i ).

[0082] Here, "x" is the horizontal axis of the two-dimensional coordinate system in which the Lissajous curve is defined, "y" is the vertical axis, and "t i ” is the acquisition time of the i-th A-line in the Lissajous scan, “A” is the scan range (amplitude), and “f A " indicates the A-line acquisition rate (scan speed, A-scan repetition rate), and n indicates the number of A-lines in each cycle in the x-direction (horizontal axis).

[0083] An example of the distribution of scan lines (scan pattern) in such an exemplary Lissajous scan is shown in FIG.

[0084] Any pair of cycles included in a Lissajous scan (in either a narrow or broad sense) intersects with each other at at least one point (particularly, two or more points). By utilizing such intersections, it becomes possible to perform registration between pairs of data collected from any pair of cycles in the Lissajous scan, and it becomes possible to implement the image construction method and the motion artifact correction method disclosed in Non-Patent Document 1 (or Non-Patent Document 2). Hereinafter, unless otherwise specified, the case where the method described in Non-Patent Document 1 is applied will be mainly described. However, the image construction method and the motion artifact correction method that can be employed in this embodiment are not limited thereto. For example, it is also possible to apply the method described in Non-Patent Document 2, or methods equivalent and / or similar to the methods described in Non-Patent Document 1 or 2.

[0085] The control information stored in the storage unit 212 is not limited to the above example. For example, the control information may include information for performing focus control (focus control parameters).

[0086] The focus control parameters are parameters that indicate the content of control over the OCT focusing drive unit 43A. Examples of focus control parameters include a parameter that indicates the focal position of the measurement arm, a parameter that indicates the moving speed of the focal position, and a parameter that indicates the moving acceleration of the focal position. The parameter that indicates the focal position is, for example, a parameter that indicates the position of the OCT focusing lens 43. The parameter that indicates the moving speed of the focal position is, for example, a parameter that indicates the moving speed of the OCT focusing lens 43. The parameter that indicates the moving acceleration of the focal position is, for example, a parameter that indicates the moving acceleration of the OCT focusing lens 43. The moving speed may or may not be constant. The same applies to the moving acceleration.

[0087] Such focus control parameters enable focus adjustment according to the shape of the fundus oculi Ef (typically a concave shape with a deep center and a shallow periphery) and aberration distribution. Focus control is performed, for example, in conjunction with scan control (repeated control of Lissajous scan). This allows motion artifacts to be corrected and high-quality images that are in focus across the entire scan range to be obtained.

[0088] <Scanning control unit 2111> The scanning control unit 2111 controls at least the optical scanner 44 based on the scanning protocol 2121. The scanning control unit 2111 may further control the light source unit 101 in cooperation with the control of the optical scanner 44 based on the scanning protocol 2121. The scanning control unit 2111 is realized by cooperation between hardware including a processor and scanning control software including the scanning protocol 2121.

[0089] <Image Data Construction Unit 220> The image data constructing unit 220 includes a processor and constructs OCT image data of the fundus Ef based on signals (sampling data) input from the data acquisition system 130. This OCT image data construction includes noise removal (noise reduction), filtering, fast Fourier transform (FFT), and the like, similar to conventional Fourier domain OCT (swept-source OCT). When another type of OCT method is adopted, the image data constructing unit 220 constructs OCT image data by performing known processing corresponding to that OCT type.

[0090] For example, the image data constructing unit 220 is configured to at least execute a process of constructing image data (A-scan image data) corresponding to each scan point (each A-line) from the sampling data. The image data constructed by the image data constructing unit 220 may be data before being imaged, and may be, for example, a signal profile (reflection profile, scattering profile) along the depth direction (z direction, optical axis direction, A-line direction).

[0091] As described above, in this embodiment, a Lissajous scan is applied to the fundus Ef. The image data constructing unit 220, together with the data processing unit 230, can construct three-dimensional image data of the fundus Ef by applying, for example, the image construction method and motion artifact correction method disclosed in Non-Patent Document 1 to the data set acquired through collection by the Lissajous scan and sampling by the data acquisition system 130.

[0092] The image data constructor 220 and / or the data processor 230 can apply rendering to the 3D image data to generate a display image. Examples of applicable rendering methods include volume rendering, surface rendering, maximum intensity projection (MIP), minimum intensity projection (MIP), and multiplanar reconstruction (MPR).

[0093] The image data constructing unit 220 and / or the data processing unit 230 can construct an en face OCT image based on the three-dimensional image data. For example, the image data constructing unit 220 and / or the data processing unit 230 can construct projection data by projecting the three-dimensional image data in the z direction (A-line direction, optical axis direction, depth direction). The image data constructing unit 220 and / or the data processing unit 230 can also construct a shadowgram by projecting partial three-dimensional image data, which is a part of the three-dimensional image data, in the z direction. This partial three-dimensional image data may be set using, for example, any segmentation method. Segmentation is a process of identifying a partial region in an image. In this example, segmentation can be performed to identify an image region corresponding to one or more tissues (regions) of the fundus oculi Ef.

[0094] The segmentation in the embodiment is not limited to the shadowgram construction process, and any known technique such as threshold processing, edge detection, filtering, machine learning (e.g., semantic segmentation) can be used.

[0095] The ophthalmic examination apparatus 1 may be capable of performing OCT angiography (OCT-angiography). OCT angiography is an imaging technique that constructs an image in which blood vessels are emphasized (see, for example, Non-Patent Document 2 and JP2015-515894A). Generally, fundus tissue (structure) does not change over a short period of time, but blood flow inside blood vessels also changes over a short period of time. OCT angiography generates an image by emphasizing areas where such temporal changes exist (blood flow signals). Note that OCT angiography is also called OCT motion contrast imaging. Furthermore, images obtained by OCT angiography are called angiographic images, angiograms, motion contrast images, etc.

[0096] When OCT angiography is possible, the ophthalmic examination apparatus 1 repeatedly scans the same region of the fundus Ef a predetermined number of times. For example, the ophthalmic examination apparatus 1 repeatedly executes the above-described scanning control (repeated control of Lissajous scanning) a predetermined number of times. As a result, multiple pieces of three-dimensional data (time-series three-dimensional data sets) are collected by the data acquisition system 130 from the application region of the Lissajous scan. The image data construction unit 220 and / or the data processing unit 230 can construct a motion contrast image from this three-dimensional data set. This motion contrast image is an angiographic image in which the temporal change in the interference signal caused by the blood flow in the fundus Ef is emphasized. This angiographic image is three-dimensional angiographic image data that represents the three-dimensional distribution of blood vessels in the fundus Ef.

[0097] The image data constructing unit 220 and / or the data processing unit 230 can construct any two-dimensional angiographic image data and / or any pseudo three-dimensional angiographic image data from this three-dimensional angiographic image data. For example, the image data constructing unit 220 and / or the data processing unit 230 can construct two-dimensional angiographic image data representing any cross section of the fundus Ef by applying multiplanar reconstruction to the three-dimensional angiographic image data. Furthermore, the image data constructing unit 220 and / or the data processing unit 230 can construct frontal angiographic image data of the fundus Ef by applying projection imaging or shadowgramming to the three-dimensional angiographic image data.

[0098] In this embodiment, the image data constructor 220 and / or the data processor 230 construct multiple strips from data collected by the data acquisition system 130. As described in Non-Patent Document 1, the image data constructor 220 and / or the data processor 230 divide the volume (three-dimensional data) collected by the Lissajous scan into multiple subvolumes without relatively large intervening motion, and construct a front projection image of each subvolume. This front projection image is a strip. Motion artifacts (in the x- and y-directions) can be corrected by applying registration and merging processes to the multiple strips obtained in this manner. As will be described later, in this example, the registration and merging processes are performed by the data processor 230.

[0099] The image data constructing unit 220 is realized by cooperation between hardware including a processor and image constructing software. In some exemplary embodiments, the image data constructing unit 220 and the data processing unit 230 may be integrally configured.

[0100] <Data processing unit 230> The data processing unit 230 includes a processor and applies various types of data processing to images of the subject's eye E. For example, the data processing unit 230 is realized by cooperation between hardware including a processor and data processing software. The same is true for each functional element included in the data processing unit 230. The data processing performed by the data processing unit 230 is not limited to processing images of the subject's eye E, and may be any data processing. As shown in FIG. 4B , the data processing unit 230 of this embodiment includes an image selection unit 225 and an image data processing unit 231.

[0101] The data processing unit 230 may be configured to perform registration between two images acquired of the fundus oculi Ef. For example, the data processing unit 230 may be configured to perform registration between three-dimensional image data acquired by OCT and a front image acquired by the fundus camera unit 2. The data processing unit 230 may also be configured to perform registration between two OCT images acquired by OCT. The data processing unit 230 may also be configured to perform registration between two front images acquired by the fundus camera unit 2. The data processing unit 230 may also be configured to apply registration to the analysis results of the OCT images and the analysis results of the front images. These registrations can be performed by known methods, and include, for example, feature point extraction and affine transformation.

[0102] The data processor 230 is configured to process a data set obtained using a Lissajous scan. As described above, the image data constructor 220 (and / or the data processor 230) constructs, for example, strips from data collected by a Lissajous scan. The data processor 230 is configured to construct a motion artifact-corrected image by applying registration and merging procedures to the strips.

[0103] As described above, Lissajous scanning collects sample data redundantly, so any two strips have an intersection area (common area, overlap area, or overlapping area). Similar to the technique described in Non-Patent Document 1, the data processing unit 230 may be configured to register the strips using the overlap between the strips. For example, the data processing unit 230 may first order the multiple strips constructed by the image data construction unit 220 according to size (e.g., area) (first to Nth strips) and designate the largest strip, the first strip, as the initial reference strip. The initial reference strip is an example of an initial reference image. Next, the data processing unit 230 registers the second strip using the first strip as a reference and merges (combines) the first and second strips. The data processing unit 230 registers the third strip using the resulting merged strip as a reference strip and merges the merged strip with the third strip. By performing such registration and merging processes in sequence in the above order, the first to Nth strips are aligned and joined together, and an image in which motion artifacts have been corrected is obtained.

[0104] In conventional techniques, such recursive registration (and merging processes) are performed based on only a single initial reference image, which may result in a deterioration in the quality of the final image, as described above. In contrast, one feature of this embodiment is that multiple recursive registrations (and merging processes) are performed, each based on a plurality of initial reference images, resulting in an advantage of improved quality in the final image. An example of the configuration of this embodiment for achieving these features and advantages is described below.

[0105] <Image selection unit 225> The ophthalmic examination apparatus 1 of this embodiment performs redundant data collection by Lissajous scanning and generates an image set based on the data set acquired thereby. In this embodiment, the image set includes multiple strips generated by the image data constructing unit 220 (or the data processing unit 230). As described above, the multiple strips have positional redundancy. In other words, any two strips (any strip pair) have positional redundancy in their intersection region.

[0106] The image selection unit 225 selects two or more strips from the generated plurality of strips. Each selected strip is used as a registration reference for adjusting the relative positions of the image set (plurality of strips). In this embodiment, each selected strip is used as an initial reference strip in recursive registration. That is, each selected strip is used as a reference image in the first registration of the registrations repeatedly performed in the recursive registration.

[0107] In this manner, in this embodiment, multiple recursive registrations are performed using multiple initial reference strips selected by the image selection unit 225. Note that in the conventional technology, a single recursive registration is performed using a single initial reference strip.

[0108] Below are described some examples of the initial reference image selection process executed by the image selection unit 225. Note that the process executed by the image selection unit 225 is not limited to these examples and may be any process applicable to selecting multiple initial reference images from a redundant image set.

[0109] A first example of the initial reference image selection process will be described. The image selector 225 capable of executing the process of this example is configured to select multiple initial reference images (multiple initial reference strips) from an image set (multiple strips) based on the time axis of optical scanning (Lissajous scanning).

[0110] The optical scanning of this embodiment sequentially scans multiple trajectories, which can be considered to be arranged in time series, and the multiple images (image sets) corresponding to the multiple trajectories can also be considered to be arranged in time series. In particular, Lissajous scanning is a scanning mode that sequentially scans multiple cycles, which can be considered to be arranged in time series, and the multiple cycles can also be considered to be arranged in time series. As can be seen from equations (9) and (10) in Non-Patent Document 1, a typical Lissajous scan is parameterized by a time parameter. The same is true for general optical scanning. The first example relates to selection of an initial reference image based on the time axis of such optical scanning.

[0111] A specific example of initial reference image selection based on the time axis of optical scanning will be described with reference to Fig. 6. The coordinate axis t in Fig. 6 represents the time axis. K sections A1, A2, ..., AK are set on the time axis t in chronological order. Here, K is an integer equal to or greater than 2. The left endpoint TS of the first section A1 in time corresponds to the scan start time, and the right endpoint TE of the last section AK in time corresponds to the scan end time.

[0112] The image group (strip group) constructed from the data set collected during the period corresponding to section A1 is designated by symbol B1. n1 The image group (strip group) constructed from the data set collected during the period corresponding to section A2 is represented by B2. n2 The image group (strip group) constructed from the data set collected during the period corresponding to the section AK is represented by BK nK Here, n1 = 1, 2, . . . , N1, n2 = 1, 2, . . . , N2, and nK = 1, 2, . . . , NK, where N1, N2, . . . , NK are all positive integers (typically integers greater than or equal to 2). If the number of images (strips) obtained by Lissajous scanning is N, then N1 + N2 + . . . + NK = N.

[0113] The K intervals A1, A2, ..., AK may or may not be equal in length. In other words, the period from the scan start time TS to the scan end time TE may or may not be divided into intervals of equal length. Furthermore, N1, N2, ..., NK may or may not be equal in length.

[0114] The image selection unit 225 of this example first selects N strips B11 to BK obtained by Lissajous scanning. NK The image set consisting of the above is divided into K subsets (N1 strips B11 to B11) corresponding to K sections A1 to A K of the time axis t of the optical scanning. N1 ;N2 strips B21~B2 N2 NK strips BK1 to BK NK ) into

[0115] Next, the image selection unit 225 of this example selects a strip from each of the K subsets. Note that strips may be selected from all of the K subsets, or strips may not be selected from any of the K subsets. Furthermore, any number of strips may be selected from one subset. Furthermore, the same number of strips may or may not be selected from each of the K subsets. In any case, the image selection unit 225 of this example selects two or more strips from the N strips arranged in time series and divided into the K subsets. Each selected strip is used as an initial reference image.

[0116] The image selector 225 in this example may be configured to select a strip from each of the K subsets based on the size of the strip (image size, e.g., area). That is, the criteria for selecting a strip may include image size.

[0117] For example, the image selection unit 225 of this example may be configured to select, from each of the K subsets, the strip with the largest size in that subset (the strip with the largest image size in that subset). Note that only the strip with the largest size in the subset may be selected, or two or more strips including the strip with the largest size may be selected.

[0118] The image selection unit 225 of this example may be configured to, for each of the K subsets, order all or some of the strips included in that subset based on image size. For example, the image selection unit 225 of this example includes a process for determining the image size of each strip, a process for comparing the determined image sizes, and a process for assigning an order to the strips based on the result of the image size comparison. The image selection unit 225 of this example may further be configured to select a strip from each of the K subsets based on the order assigned to the strips in that subset. This allows one or more strips to be selected from the subset in accordance with the order according to image size.

[0119] As will be described in detail later, the image selection unit 225 of this example may be configured to select another strip from the same subset based on the result of registration performed using a strip selected from the subset as an initial reference image. For example, the image selection unit 225 of this example can select one strip (first image) from one of the K subsets based on the order in which the strips in the subset are assigned. After registration based on the selected strip is performed, the image selection unit 225 of this example can determine whether the registration is successful. If the registration is determined to be successful, the registration based on this subset is determined to be complete. On the other hand, if the registration is determined to be unsuccessful, the image selection unit 225 of this example can select another strip (second image) from the subset based on the order in which the strips in the subset are assigned.

[0120] A second example of the initial reference image selection process will now be described. The image selector 225 capable of executing the process of this example is configured to select multiple initial reference images (multiple initial reference strips) from an image set (multiple strips) based on the application area of ​​optical scanning (Lissajous scanning). In contrast to the first example in which the image set is divided temporally, this example can be said to divide the image set spatially.

[0121] 7, 8 and 8B, a specific example of initial reference image selection based on the application area of ​​optical scanning will be described.

[0122] In the specific example of Fig. 7, the symbol C indicates the application area of ​​Lissajous scanning (for example, the scan area shown in Fig. 5). In this specific example, the application area C is divided into two equal parts in the x direction (left and right direction) and in the y direction (up and down direction), thereby forming four sub-areas C1, C2, C3, and C4.

[0123] In this example, the image selection unit 225 divides the image set (multiple strips) obtained by Lissajous scanning into four subsets corresponding to the four subareas C1 to C4. The subset corresponding to subarea C1 includes images in subarea C1 (for example, a portion of the strip (partial strip)). The same applies to the subsets corresponding to subareas C2 to C4.

[0124] Furthermore, the image selector 225 of this example selects multiple images (e.g., multiple partial strips) from the image set (multiple strips) by selecting an image (e.g., multiple partial strips) from each of the four subsets corresponding to the four sub-areas C1 to C4. Each selected partial strip is used as an initial reference image.

[0125] In this example, the image selector 225 may be configured to select a partial strip from each of the four subsets based on the size (image size, e.g., area) of the partial strip. That is, the selection criteria for the partial strip may include the image size.

[0126] For example, the image selection unit 225 in this example may be configured to select, from each of the four subsets, the partial strip with the largest size in that subset (the partial strip with the largest image size in that subset). Note that it may also be possible to select only the partial strip with the largest size in the subset, or to select two or more partial strips including the partial strip with the largest size.

[0127] The image selection unit 225 of this example may be configured to, for each of the four subsets, order all or some of the partial strips included in that subset based on image size. For example, the image selection unit 225 of this example includes a process for determining the image size of each partial strip, a process for comparing the determined image sizes, and a process for assigning an order to the partial strips based on the result of the image size comparison. The image selection unit 225 of this example may further be configured to select a partial strip from each of the four subsets based on the order assigned to the partial strips in that subset. This allows one or more partial strips to be selected from the subset in accordance with the order according to image size.

[0128] The image selector 225 of this example may be configured to select another partial strip from the same subset based on the result of registration performed using a partial strip selected from the subset as an initial reference image. For example, the image selector 225 of this example can select one partial strip (first image) from one of the four subsets based on the order in which the partial strips in the subset are assigned. After registration based on the selected partial strip is performed, the image selector 225 of this example can determine whether the registration is successful. If the registration is determined to be successful, the registration based on this subset is determined to be complete. On the other hand, if the registration is determined to be unsuccessful, the image selector 225 of this example can select another partial strip (second image) from the subset based on the order in which the partial strips in the subset are assigned.

[0129] 7, the application area C of the Lissajous scan is divided into four subareas C1 to C4 of equal shape and size, but the manner in which the application area is divided is not limited to this. For example, the number of subareas formed by dividing the application area may be any number equal to or greater than two. Furthermore, the two or more subareas formed by dividing the application area may have different shapes and / or different sizes.

[0130] The division of the application area is not limited to the grid division as shown in the specific example of Fig. 7, and may be any division such as radial division, concentric division, etc. Furthermore, the division of the application area is not limited to geometric division such as grid division, radial division, or concentric division.

[0131] In some exemplary embodiments, the application area can be divided based on objects depicted in the application area. For example, when a first object and a second object are depicted in the application area, the application area can be divided into two or more subareas including a subarea including an image of the first object and a subarea including the second object. In the case of a Lissajous scan of the fundus Ef, the application area of ​​this Lissajous scan can be divided into two or more subareas including a subarea including an image of the optic disc and a subarea including the macula.

[0132] In some exemplary embodiments, an importance distribution can be assigned to at least a portion of the application area, and the application area can be divided into two or more subareas based on this importance distribution. For example, the application area can be divided into two or more subareas including a subarea including a highly important portion and the other subareas. In the case of a Lissajous scan of the fundus Ef, an image of the optic disc, an image of the macula, an image of the lesion, etc. can be designated as a highly important portion.

[0133] The specific examples in Figures 8A and 8B are applications to a scan mode (called convolutional Lissajous scan) disclosed in Japanese Patent Application Laid-Open No. 2021-40854. Convolutional Lissajous scan is a scan mode that combines a standard Lissajous scan as shown in Figure 5 with a shift in its application area (application area shift) (see Figure 8A). In the example of Figure 8A, the application area shift is performed along a circular trajectory, but the trajectory is not limited to this and may be any trajectory. Convolutional Lissajous scan enables both motion artifact correction and a wider angle of view (expansion of the application area) of imaging.

[0134] In the example of FIG. 8B, the symbol D indicates the application area of ​​the convolutional Lissajous scan. In this example, the application area D is divided into three equal parts in the angular direction defined around its center. By doing so, three sub-areas D1, D2, and D3 are formed.

[0135] In this example, the image selection unit 225 divides the image set (multiple strips) obtained by the Lissajous scan into three subsets corresponding to the three sub-areas D1 to D3. The subset corresponding to the sub-area D1 includes images (e.g., partial strips) in the sub-area D1. The same applies to the subsets corresponding to the sub-areas D2 and D3.

[0136] Furthermore, the image selector 225 of this example selects multiple images (e.g., multiple partial strips) from the image set (multiple strips) by selecting an image (e.g., multiple partial strips) from each of the three subsets corresponding to the three sub-areas D1 to D3. Each selected partial strip is used as an initial reference image.

[0137] The image selector 225 in this example may be configured to select a partial strip from each of the three subsets based on the size (image size, e.g., area) of the partial strip. That is, the selection criteria for the partial strip may include the image size.

[0138] For example, the image selection unit 225 in this example may be configured to select, from each of the three subsets, the partial strip with the largest size in that subset (the partial strip with the largest image size in that subset). Note that it may also be possible to select only the partial strip with the largest size in the subset, or to select two or more partial strips including the partial strip with the largest size.

[0139] The image selection unit 225 of this example may be configured to, for each of the three subsets, order all or some of the partial strips included in that subset based on image size. For example, the image selection unit 225 of this example includes a process for determining the image size of each partial strip, a process for comparing the determined image sizes, and a process for assigning an order to the partial strips based on the result of the image size comparison. The image selection unit 225 of this example may further be configured to select a partial strip from each of the three subsets based on the order assigned to the partial strips in that subset. This allows one or more partial strips to be selected from the subset in accordance with the order according to image size.

[0140] The image selection unit 225 of this example may be configured to select another partial strip from the same subset based on the result of registration performed using a partial strip selected from the subset as an initial reference image. For example, the image selection unit 225 of this example can select one partial strip (first image) from one of the three subsets based on the order in which the partial strips in this subset are assigned. After registration based on the selected partial strip is performed, the image selection unit 225 of this example can determine whether this registration is successful. If this registration is determined to be successful, registration based on this subset is determined to be complete. On the other hand, if this registration is determined to be unsuccessful, the image selection unit 225 of this example can select another partial strip (second image) from this subset based on the order in which the partial strips in this subset are assigned.

[0141] In the specific example of Figure 8B, the application area D of the Lissajous scan is divided into three equal angular parts to form three sub-areas D1 to D3, but as explained with respect to the specific example of Figure 7, the manner in which the application area is divided is not limited to this.

[0142] A third example of the initial reference image selection process will now be described. The image selection unit 225 capable of executing the process of this example is configured to order all or only some of the images included in an image set (plurality of strips) obtained by optical scanning (Lissajous scanning) based on image size, and select multiple initial reference images (plurality of initial reference strips) from the image set based on the order assigned to the images by this ordering.

[0143] In this example, instead of dividing the image set as in the first and second examples, multiple initial reference images are selected from the entire image set. Note that in some exemplary aspects, the image selector 225 may be configured to exclude one or more images that satisfy a predetermined condition from the image set acquired by optical scanning, and select multiple initial reference images from the remaining image set. This condition may be, for example, a condition related to each image (image quality, signal quality, etc.) or a condition related to the relationship between images (image correlation, positional deviation, difference in image quality, difference in signal quality, etc.).

[0144] The image selection unit 225 of this example may be configured to select another initial reference image (an image different from any of the previously selected multiple initial reference images) from the image set based on the result of registration performed using one image of multiple initial reference images selected from the image set based on the order in which the image sizes are referenced. For example, the image selection unit 225 of this example first performs ordering on the image set (multiple strips) based on the image sizes and selects multiple initial reference strips (initial reference strip group) from the image set based on the assigned order. After performing registration based on one initial reference strip from the initial reference strip group, the image selection unit 225 of this example can determine whether the registration is successful. If the registration is determined to be successful, the initial reference strip is determined to be appropriate. On the other hand, if the registration is determined to be unsuccessful, the initial reference strip is determined to be inappropriate, and the image selection unit 225 of this example newly selects a strip from the image set that is different from any of the multiple images included in the initial reference strip group based on the order assigned to the image set. This allows inappropriate strips from the initially selected initial reference strip group to be replaced with new strips, ultimately resulting in an initial reference strip group consisting of only strips suitable for motion artifact correction.

[0145] A fourth example of the initial reference image selection process will be described. In this example, an initial reference image is selected by referring to the movement (motion) of the sample when optical scanning is applied. This sample movement is a movement that causes or may cause a motion artifact. In the field of ophthalmology, the movement of the sample (examined eye E) includes eye movement, head movement, body movement, etc. This example can be combined with the first to third examples or other examples of the initial reference image selection described above. For example, the process of this example can be executed as preprocessing of any of the processes of the first to third examples.

[0146] The ophthalmic examination apparatus 1 capable of executing the processing of this example includes a motion information generating unit that generates motion information representing the motion of the subject's eye E when optical scanning is applied to the sample. The configuration of the motion information generating unit may be arbitrary, and may include, for example, any of the configurations described below. Note that the following exemplary configuration is related to eye motion, but may also be applied to other samples.

[0147] A first configuration example of the movement information generating unit detects the movement of the subject's eye E from an OCT angiography signal (OCTA signal, motion contrast signal). Non-Patent Document 2 describes that in OCT angiography using a Lissajous scan, eye movement is detected from the OCT angiography signal, and data affected by eye movement (repeated scan cycle data, repeat-cycle-set) is discarded.

[0148] When the movement information generating unit of this configuration example is applied, the ophthalmic examination apparatus 1 may be configured to, for example, perform a Lissajous scan for OCT angiography using the fundus camera unit 2, the OCT unit 100, and the control unit 210 to collect a data set, process this data set using the image data constructing unit 220 and the data processing unit 230 to generate an OCT angiography signal, detect eye movement from this OCT angiography signal using the image selecting unit 225, and exclude from candidates for the initial reference strip a strip in a period or region near the time when eye movement satisfying a predetermined condition was detected by the image selecting unit 225. This condition may be any, and may be, for example, a condition that the magnitude of eye movement is equal to or greater than a predetermined threshold and / or a condition that the frequency of eye movement occurrence is equal to or greater than a predetermined threshold.

[0149] The second configuration example of the movement information generating unit detects the movement of the subject's eye E from data other than the OCT signal. Data that can be used for movement detection in this configuration example include observation images acquired by the fundus camera unit 2 and anterior eye images acquired by the anterior eye camera disclosed in Japanese Patent Application Laid-Open No. 2013-248376. This configuration example does not require the repeated scans required for OCT angiography in the first configuration example.

[0150] When the movement information generating unit of this configuration example is applied, the ophthalmic examination apparatus 1 may be configured to, for example, acquire an image of the subject's eye E using the fundus camera unit 2 or an anterior eye camera (not shown), detect eye movement from this image using the data processing unit 230 (e.g., the image selecting unit 225), and exclude from candidates for the initial reference strip a strip in a region near a period or position near the time when eye movement satisfying a predetermined condition was detected by the image selecting unit 225. This condition may be any, and may be, for example, a condition that the magnitude of the eye movement is equal to or greater than a predetermined threshold, and / or a condition that the frequency of occurrence of eye movement is equal to or greater than a predetermined threshold.

[0151] In this way, when the ophthalmic examination apparatus 1 is equipped with a motion information generation unit that generates motion information representing the motion of the test eye E when a Lissajous scan is applied to the test eye E, the image selection unit 225 may be configured to select multiple initial reference images from the image set (multiple strips) based on the motion information generated by the motion information generation unit.

[0152] Furthermore, the image selection unit 225 may be configured to perform the processes of calculating values ​​of predetermined movement parameters from the movement information generated by the movement information generation unit, evaluating the calculated values, and selecting multiple initial reference images from the image set (multiple strips) based on the results of this evaluation.

[0153] The motion parameters used to evaluate the motion information may include at least one of the magnitude of eye movement and the frequency of eye movement. Furthermore, the image selection unit 225 can compare the calculated value of such motion parameter with a predetermined threshold. If it is determined that the value of the motion parameter is equal to or greater than the threshold, the image selection unit 225 can identify a period corresponding to the value of the motion parameter, identify images (strips) based on data collected by the Lissajous scan during the identified period, and select multiple initial reference images from a subset in which one or more identified images are excluded from the image set. Here, the period corresponding to the value of the motion parameter may be, for example, a period of a predetermined length including the time when eye movement of a magnitude equal to or greater than the threshold occurred and / or a period of a predetermined length including the time when eye movement of a frequency equal to or greater than the threshold occurred.

[0154] According to a fourth example of the initial reference image selection process, an eye movement detection function such as tracking can be combined with the initial reference image selection. This combination allows the registration method to be switched near the occurrence of eye movements such as saccades. For example, data near the occurrence of eye movements can be excluded from registration. Furthermore, images (strips) near the occurrence of eye movements such as saccades can be excluded from the candidates for the initial reference image. Furthermore, images (strips) near the occurrence of frequent eye movements such as saccades can be excluded from the candidates for the initial reference image.

[0155] Although the above-described exemplary initial reference image selection processes have been described with reference to image size, the information (parameters) referenced in initial reference image selection are not limited to image size. For example, embodiments may include an aspect in which ordering based on an arbitrary image quality evaluation index is applied to a set of images (plurality of strips), an aspect in which ordering based on the amount of an arbitrary feature amount of the images (strips) is applied to a set of images (plurality of strips), an aspect in which ordering based on an index that combines an arbitrary image quality evaluation index and an arbitrary image feature amount is applied to a set of images (plurality of strips), an aspect in which ordering based on an index that combines an arbitrary image quality evaluation index and / or an arbitrary image feature amount and image size is applied to a set of images (plurality of strips), or an aspect in which ordering based on an index other than these is applied to a set of images (plurality of strips).

[0156] <Image data processing unit 231> The following describes the configuration and background of the image data processing unit 231. In recursive registration for motion artifact correction, a cross-correlation function is used to determine the relative position between a reference strip and other strips. However, since strips can have arbitrary shapes, correlation calculations between strips are performed using a mask that treats each strip as an image of a specific shape (for example, a square shape) (see Appendix A of Non-Patent Document 1).

[0157] As mentioned above, the strip is an intensity image of a predetermined gradation, and the mask image is a binary image, so the difference between the absolute values ​​of the pixel values ​​of the strip and the mask becomes large, and there is a risk that the effect of the mask in the correlation calculation using the strip and the mask will be ignored and an accurate correlation coefficient will not be obtained. In particular, when single-precision floating-point (float type) calculations are used to calculate the correlation coefficient between strips from the perspective of cost, etc., this problem becomes more pronounced due to the influence of rounding errors caused by the small number of significant digits.

[0158] In addition to dealing with the rounding error problem, the image data processing unit 231 includes the elements shown in Fig. 4C to adjust the relative positions of the image set based on a plurality of initial reference images. Note that the configuration for adjusting the relative positions of the image set based on a plurality of initial reference images is not limited to the configuration shown in Fig. 4C, and may be any configuration capable of performing relative position adjustment according to the exemplary aspects described below or similar relative position adjustments.

[0159] 4C , the image data processing unit 231 processes the image data constructed by the image data construction unit 220. As shown in FIG. 4C , the image data processing unit 231 includes a mask image generation unit 2311, a range adjustment unit 2312, a composite image generation unit 2313, a cross-correlation function calculation unit 2314, a correlation coefficient calculation unit 2315, an xy shift amount calculation unit 2316, a registration unit 2317, a merge processing unit 2318, a z shift amount calculation unit 232, a shift amount recording unit 233, an adjustment amount calculation unit 234, and an image data correction unit 235.

[0160] In the example described below, the image data processing unit 231 is configured to designate one of any two strips as a reference strip and to register the other strip (registering strip) with respect to this reference strip.

[0161] <Mask image generation unit 2311> The mask image generator 2311 generates a reference mask image corresponding to the reference strip and a target mask image corresponding to the target strip. Some examples of mask images are described below, but the present invention is not limited to these.

[0162] The contour shape of the mask image is, for example, rectangular, typically square. The shape of the reference mask image and the shape of the target mask image may be the same. Furthermore, the size of the reference mask image and the size of the target mask image may be the same.

[0163] The range of pixel values ​​of the mask image is set to be included in the closed interval [0,1], for example. Typically, the mask image may be a binary image with pixel values ​​of 0 or 1. As a specific example, the mask image is a binary image in which the pixel value in the area corresponding to the domain of the strip is 1 and the other pixels are 0. In other words, as shown in equation (20) of Non-Patent Document 1, the pixel value of the exemplary mask image is 1 in the image area of ​​the corresponding strip and 0 in the other areas.

[0164] <Range Adjustment Unit 2312> The range adjuster 2312 is configured to adjust the pixel value range of the reference strip and the pixel value range of the target strip based on the pixel value range of the reference mask image and the pixel value range of the target mask image. Typically, the range adjuster 2312 adjusts the pixel value range of the reference strip and the pixel value range of the target strip so as to reduce the difference between the pixel value range of the strip and the pixel value range of the mask image.

[0165] In general, the range adjuster 2312 adjusts the pixel value range of the reference strip, the pixel value range of the target strip, and the pixel value range of the mask image relatively. In a typical example, the reference mask image applied to the reference strip and the target mask image applied to the target strip are different, and the range adjuster 2312 adjusts the pixel value range of the reference strip, the pixel value range of the target strip, the pixel value range of the reference mask image, and the pixel value range of the target strip relatively.

[0166] The range adjustment unit 2312 is configured to adjust the range of pixel values ​​of the reference strip and the range of pixel values ​​of the reference mask image so as to reduce the range of pixel values ​​of the reference strip and the range of pixel values ​​of the reference mask image, and to relatively adjust the range of pixel values ​​of the target strip and the range of pixel values ​​of the target mask image so as to reduce the range of pixel values ​​of the target strip and the range of pixel values ​​of the target mask image.

[0167] The range adjustment unit 2312 may be configured to adjust the pixel value range of the strip and the pixel value range of the mask image so that the pixel value range of one of the strip and the mask image matches the pixel value range of the other. For example, the range adjustment unit 2312 may be configured to adjust the pixel value range of the reference strip and the pixel value range of the reference mask image so that the pixel value range of the reference strip and the pixel value range of the reference mask image match, and to relatively adjust the pixel value range of the target strip and the pixel value range of the target mask image so that the pixel value range of the target strip and the pixel value range of the target mask image match.

[0168] For example, the range adjuster 2312 is configured to normalize the range of pixel values ​​of the strip according to the range of pixel values ​​of the mask image, some examples of this normalization (standardization) are described below, but are not limited to these.

[0169] A first example of normalization will be described. When the range of pixel values ​​of the mask image is included in the closed interval [0, 1], the range adjustment unit 2312 divides the value of each pixel in the strip by the maximum pixel value in the strip. In this example, the range adjustment unit 2312 first compares the values ​​of all pixels in the strip to identify the maximum value (maximum pixel value), and then divides the value of each pixel in the strip by the maximum pixel value. This causes the range of pixel values ​​of the strip to match the closed interval [0, 1], which is the same as the range of pixel values ​​of the mask image.

[0170] A second example of normalization will be described. When the range of pixel values ​​of the mask image is included in the closed interval [0, 1], the range adjustment unit 2312 divides the value of each pixel of the strip by the maximum value of the range of pixel values ​​of this strip. The range of pixel values ​​of the strip is set in advance, and the range adjustment unit 2312 divides the value of each pixel of the strip by the upper limit (maximum value) of this range. In this example, the range of pixel values ​​of the strip is also matched to the same closed interval [0, 1] as the range of pixel values ​​of the mask image.

[0171] The processing performed by the range adjustment unit 2312 can reduce the difference between the absolute values ​​of the pixel values ​​of the strip and the absolute values ​​of the pixel values ​​of the mask, so that the effect of the mask is not ignored in the correlation calculation using the strip and the mask, and it becomes possible to accurately calculate the correlation coefficient. In particular, even when single-precision floating-point (float) calculation is used, it is possible to eliminate (reduce) the influence of rounding errors caused by a small number of significant digits.

[0172] In this disclosure, we mainly describe examples of changing only the pixel value range of the strip, but it is also possible to change only the pixel value range of the mask image, or to change both the pixel value range of the strip and the pixel value range of the mask image.

[0173] <Synthetic image generation unit 2313> For the strips and mask images to which the pixel value range adjustment by the range adjustment unit 2312 has been applied, the composite image generation unit 2313 generates two composite images by combining the mask image with each of the reference strip and the target strip. Typically, the composite image generation unit 2313 combines the reference mask image with the reference strip to generate a reference composite image, and combines the target mask image with the target strip to generate a target composite image.

[0174] The process of combining the strip and the mask image may be performed in the same manner as in Non-Patent Document 1. However, unlike the method in Non-Patent Document 1, in this embodiment, an adjustment is made to reduce the difference between the pixel value range of the strip and the pixel value range of the mask image. Typically, the pixel value ranges of the reference strip and the target strip are normalized to match the pixel value ranges of the reference mask image and the target mask image.

[0175] For example, in the same manner as equation (19) in Non-Patent Document 1, the composite image generation unit 2313 may be configured to embed a strip with a normalized pixel value range into an image of the same size and shape as the mask image.

[0176] Furthermore, the composite image generation unit 2313 generates a composite image of the embedded image of the strip and the mask image. This composite image is based on the "f'(r)m f (r)" (page 1800, line 2, etc.), but as mentioned above, the value of the embedded image of the strip is different from that of equation (19).

[0177] <Cross-correlation function calculation unit 2314> The cross-correlation function calculation unit 2314 calculates multiple cross-correlation functions based on the two composite images generated by the composite image generation unit 2313. The calculation of the cross-correlation functions is performed in the same manner as in Non-Patent Document 1. For example, the cross-correlation function calculation unit 2314 calculates six cross-correlation functions (image cross-correlation) included in equation (33) in Non-Patent Document 1 based on a reference composite image generated from a reference strip and a reference mask image, and a target composite image generated from a target strip and a target mask image.

[0178] <Correlation coefficient calculation unit 2315> The correlation coefficient calculation unit 2315 calculates a correlation coefficient based on the multiple cross-correlation functions calculated by the cross-correlation function calculation unit 2314. This calculation follows equation (33) in Non-Patent Document 1.

[0179] <xy shift amount calculation unit 2316> Based on the correlation coefficient calculated by the correlation coefficient calculation unit 2315, the xy shift amount calculation unit 2316 calculates the shift amount in the xy direction (lateral direction, horizontal direction) between the reference strip and the target strip.

[0180] The operation for calculating the xy direction shift amount includes, for example, an operation corresponding to "rough lateral motion correction" (page 1787) in Non-Patent Document 1. In this case, the xy shift amount calculation unit 2316 is configured to estimate the xy direction shift amount by obtaining the maximum value of the cross-correlation function.

[0181] Furthermore, the xy shift amount calculation unit 2316 may be configured to execute an operation corresponding to "fine lateral motion correction" (page 1789) in Non-Patent Document 1 in order to calculate a small shift amount in the lateral direction caused by eye movements such as slow drift and tremor.

[0182] The xy direction shift amount data obtained by the xy shift amount calculation unit 2316 is sent to the shift amount recording unit 233 and recorded together with predetermined information (ancillary information). Examples of this ancillary information include an identifier for the xy direction shift amount data, the correlation coefficient corresponding to the xy direction shift amount, and the like.

[0183] The identifier for the xy direction shift amount data may include, for example, any of the following: time expressed by the time axis of optical scanning (residual scan) (see, for example, FIG. 6); number based on the cycle ordering (cycle number); number based on the scan order (scan number); number based on the strip ordering (strip number); number based on the sub-volume ordering (sub-volume number); position expressed in the xy coordinate system (pixel position, scan position, etc.). All of these pieces of information are equivalent. That is, for any two of these pieces of information, it is possible to convert one piece of information into the other.

[0184] The identifier of the xy-direction shift amount data may include an identifier (common identifier) ​​common to multiple xy-direction shift amount data. For example, for each of multiple initial reference strips specified by the image selection unit 225, a common identifier may be assigned to multiple xy-direction shift amount data sequentially obtained in the recursive registration performed based on the initial reference strip. This common identifier makes it possible to identify and process multiple xy-direction shift amount data corresponding to each initial reference strip (i.e., multiple xy-direction shift amount data corresponding to each recursive registration). The common identifier may be provided as information separate from the identifiers (individual identifiers) of each xy-direction shift amount data, or the common identifier and the individual identifier may be formed as integrated information.

[0185] <Registration Department 2317> The registration unit 2317 performs registration in the lateral direction based on the amount of lateral shift (xy direction shift amount) calculated by the xy shift amount calculation unit 2316. For example, this registration includes processing equivalent to "rough lateral motion correction" (page 1787) in Non-Patent Document 1. The registration unit 2317 can perform registration between the reference strip and the target strip so as to cancel out the amount of lateral shift calculated by the xy shift amount calculation unit 2316.

[0186] When the xy shift amount calculation unit 2316 performs a calculation equivalent to "fine lateral motion correction" (page 1789) in Non-Patent Document 1, the registration unit 2317 can remove small lateral motion artifacts between the reference strip and the target strip by performing a registration equivalent to "fine lateral motion correction" (page 1789) in Non-Patent Document 1.

[0187] <Merge processing unit 2318> The merge processing unit 2318 constructs a merged image of a reference strip and a target strip whose relative positions have been adjusted by the registration unit 2317. This process is also executed in the same manner as the method described in Non-Patent Document 1.

[0188] As described above, the image data processing unit 231 sequentially executes the above-described series of processes on the plurality of strips constructed by the image data construction unit 220 in the order according to their sizes. As a result, a merged image in which lateral motion artifacts are corrected is obtained from the plurality of strips constructed by the image data construction unit 220. This merged image is typically an image representing the entire range to which a resized scan is applied.

[0189] As described above, the plurality of strips are a plurality of frontal projection images based on a plurality of sub-volumes obtained by dividing a volume (three-dimensional data) collected by a resized scan. Therefore, the merged image constructed from the plurality of strips by the image data processing unit 231 provides a plurality of sub-volumes (and their merged images) whose lateral positions have been adjusted. In this example, the registration and merge processing in the depth direction orthogonal to the lateral direction are executed by the z-shift amount calculation unit 232 and the image data correction unit 235 (described later).

[0190] <z-shift amount calculation unit 232> The z-shift amount calculation unit 232 calculates the shift amount in the z direction (A-line direction, optical axis direction, depth direction) between a plurality of sub-volumes obtained by dividing a volume collected by a resized scan (that is, a plurality of sub-volumes that are the source of the plurality of strips).

[0191] In this example, the image data processing unit 231 performs position adjustment in the x and y directions (lateral direction), and then the z shift amount calculation unit 232 and the image data correction unit 235 perform position adjustment in the z direction. That is, the z shift amount calculation unit 232 in this example is configured to calculate the shift amount in the z direction using data obtained by the image data processing unit 231 (for example, x and y direction shift amount data, merged image).

[0192] An example of processing executed by the z-shift amount calculation unit 232 will be described. Similar to the pair of strips (reference strip and target strip) considered in calculating the shift amounts in the x and y directions, a pair of sub-volumes is also considered in calculating the shift amounts in the z direction. The pair of sub-volumes may be two sub-volumes corresponding to the reference strip and target strip considered in calculating the shift amounts in the x and y directions, and these sub-volumes are referred to as the reference sub-volume and the target sub-volume, respectively.

[0193] Based on the result of the xy-direction registration between the corresponding reference strips and target strips, the reference sub-volumes and target sub-volumes are aligned in the xy-direction. Also, as mentioned above, any two strips in a Lissajous scan overlap (intersect) at four points, and therefore any two sub-volumes also overlap (intersect) at four points.

[0194] The z-shift amount calculation unit 232 first identifies an intersection area (common area) between the reference sub-volume and the target sub-volume whose positions have been adjusted in the x and y directions. Each intersection area is three-dimensional image data.

[0195] Next, the z-shift amount calculation unit 232 sets a cross section in the identified intersection area. This cross section is, for example, a plane formed by an arbitrary axis (for example, the x-axis, the y-axis, or an axis obliquely intersecting both the x-axis and the y-axis) in the xy plane and the z-axis. Note that the cross section is not limited to a plane, and may be a curved surface, etc.

[0196] Next, the z-shift amount calculation unit 232 constructs an image of the set cross section from the reference sub-volume, and also constructs an image of the same cross section from the target sub-volume. The cross-sectional image constructed from the reference sub-volume is called the reference cross-sectional image, and the cross-sectional image constructed from the target sub-volume is called the target cross-sectional image. The reference cross-sectional image and the target cross-sectional image are images that represent the same cross section in the intersection region between the reference sub-volume and the target sub-volume that have been registered in the x- and y-directions.

[0197] Next, the z-shift amount calculation unit 232 analyzes the reference cross-sectional image to identify an image of a predetermined portion of the subject's eye E, and analyzes the target cross-sectional image to identify an image of the same portion. This portion may be any portion, and may be, for example, the surface of the fundus Ef (the retinal surface, the internal limiting membrane, or the boundary between the retina and the vitreous body). Furthermore, if an artificial object has been implanted in the subject's eye E, an image of this artificial object may be identified. The analysis for identifying the image of the predetermined portion may include, for example, segmentation.

[0198] Next, the z-shift amount calculation unit 232 calculates the z-coordinate of an image (reference image) of a predetermined region identified from the reference cross-sectional image, and calculates the z-coordinate of an image (target image) of a predetermined region identified from the target cross-sectional image. Typically, the image of the predetermined region consists of multiple pixels, and the z-coordinates of these pixels are not constant. For example, the global shape of the retinal surface is generally a curved shape convex in the +z direction, and images of local areas (examples of the reference image and target image) are at least partially tilted with respect to the z-axis. The z-shift amount calculation unit 232 can calculate the z-coordinate of the image of the predetermined region based on at least one pixel of a group of pixels constituting the image of the predetermined region. For example, a statistic of the z-coordinates can be calculated from the group of pixels, and this statistic can be used as the z-coordinate of the image. This statistic can be any representative value (summary statistic), such as a maximum value, minimum value, mean value, median, mode, or quantile. On the other hand, if the image of the predetermined region consists of a single pixel, the z-coordinate of this pixel can be used as the z-coordinate of the predetermined region.

[0199] Next, the z-shift amount calculation unit 232 calculates the difference between the z-coordinate of the reference image and the z-coordinate of the target image (the shift amount in the z direction). In some exemplary aspects, processing control can be performed so that correction processing is performed by the image data correction unit 235 when the z-direction shift amount exceeds a threshold. This threshold is, for example, zero or a positive value, and may be a preset fixed value or a variable value.

[0200] In the above example, the z-direction shift amount is calculated by constructing two-dimensional image data (cross-sectional images) from three-dimensional image data (sub-volumes), but in some exemplary aspects, the z-direction shift amount may be calculated from three-dimensional image data without constructing two-dimensional image data. In this case, the same processing as that performed on the pair of reference cross-sectional images and target cross-sectional images can be performed on the pair of reference sub-volumes and target sub-volumes.

[0201] In the above example, two-dimensional image data (cross-sectional image) is constructed from three-dimensional image data (subvolume), and an image of a specified area is identified from this two-dimensional image data to determine the amount of shift in the z direction. However, in some exemplary embodiments, an image of a specified area may be identified from the three-dimensional image data, and two-dimensional image data of the image of this specified area may be constructed to determine the amount of shift in the z direction.

[0202] The z-shift amount calculation unit 232 may be configured to calculate the z-direction shift amount between the reference subvolume and the target subvolume based on a correlation coefficient calculated based on a pair of the reference subvolume and the target subvolume. The method for calculating the correlation coefficient between the subvolumes may be the same as the correlation coefficient calculation method executed by the correlation coefficient calculation unit 2315, for example.

[0203] The method for calculating the z-direction shift amount is not limited to the above-mentioned methods, and may be any method applicable to a pair of a reference subvolume and a target subvolume, or may be any method applicable to a pair of an object other than a subvolume. An example of an object other than a subvolume is a rendered image of the subvolume (an image having a dimension in the z direction, such as an image projected in the x direction or y direction).

[0204] In some exemplary aspects, the z-direction shift amount data calculated by the z-shift amount calculation unit 232 is sent to and recorded in the shift amount recording unit 233 together with predetermined information (accompanying information). Examples of this accompanying information include an identifier for the z-direction shift amount data, a value or amount (e.g., a correlation coefficient) calculated during the calculation for calculating the z-direction shift amount, and the like.

[0205] Similar to the individual identifier of the xy-direction shift amount data, the individual identifier of the z-direction shift amount data may include any of the following: time represented by the time axis of the optical scan (Lissajous scan) (see, for example, FIG. 6); a number based on the ordering of cycles (cycle number); a number based on the order of scans (scan number); a number based on the ordering of strips (strip number); a number based on the ordering of subvolumes (subvolume number); or a position represented in the xy coordinate system (pixel position, scan position, etc.). Also, similar to the identifier of the xy-direction shift amount data, the identifier of the z-direction shift amount data may include an identifier (common identifier) ​​common to multiple pieces of z-direction shift amount data.

[0206] <Shift amount recording unit 233> The shift amount recording unit 233 records the xy direction shift amount data calculated by the xy shift amount calculation unit 2316 and its accompanying information. The accompanying information in this embodiment includes identifiers (e.g., individual identifiers and common identifiers) of the xy direction shift amount data described above. The shift amount recording unit 233 includes, for example, a storage device and a processor that writes data to and reads data from the storage device.

[0207] As described above, the xy shift amount calculation unit 2316 of this embodiment generates xy direction shift amount data for each pair of a reference strip and a target strip. In some exemplary aspects, the shift amount recording unit 233 collects and records xy direction shift amount data corresponding to all pairs of a reference strip and a target strip. In some exemplary aspects, the shift amount recording unit 233 collects and records xy direction shift amount data corresponding to a portion of all pairs of a reference strip and a target strip.

[0208] In some exemplary aspects, the shift amount recording unit 233 records the z-direction shift amount data calculated by the z-shift amount calculation unit 232 and its accompanying information. The accompanying information in this embodiment includes the identifiers of the z-direction shift amount data described above (e.g., individual identifiers and common identifiers). When a configuration in which the shift amount recording unit 233 records the z-direction shift amount data is applied, an arrow indicating data transmission from the z-shift amount calculation unit 232 to the shift amount recording unit 233 is added between the z-shift amount calculation unit 232 and the shift amount recording unit 233 in the drawing of FIG. 4C.

[0209] In some exemplary embodiments, the shift amount recording unit 233 collects and records z-direction shift amount data corresponding to all pairs of reference subvolumes and target subvolumes, and in some exemplary embodiments, the shift amount recording unit 233 collects and records z-direction shift amount data corresponding to some of all pairs of reference subvolumes and target subvolumes.

[0210] Note that there is a correspondence between the strips and the sub-volumes, and therefore there is a correspondence between the pair of the reference strip and the target strip and the reference sub-volume and the target sub-volume. Therefore, in an aspect in which the shift amount recording unit 233 records both the xy direction shift amount data and the z direction shift amount data, it is possible to associate the xy direction shift amount data with the z direction shift amount data.

[0211] <Adjustment amount calculation unit 234> The adjustment amount calculation unit 234 is configured to calculate an adjustment amount for adjusting the relative position of the image set based on the data recorded by the shift amount recording unit 233 .

[0212] The adjustment amount calculation unit 234 in some exemplary embodiments calculates adjustment amounts for adjusting the relative positions of multiple strips based on the x and y direction shift amount data recorded by the shift amount recording unit 233. The x and y direction shift amount data recorded by the shift amount recording unit 233 is assigned the above-mentioned individual identifier and common identifier. Each common identifier represents a set of x and y direction shift amount data (x and y direction shift amount data set) obtained from one initial reference strip (i.e., one recursive registration), and each individual identifier represents the position (e.g., temporal position or spatial position) of each x and y direction shift amount data in the set.

[0213] One set of xy-direction shift amount data is information (relative position information) representing the relative positions between multiple strips included in an image set, and is substantially the same as the relative position information obtained by a conventional method (e.g., the method of Cited Document 1). However, this embodiment differs from the conventional method in that multiple pieces of relative position information are obtained by performing multiple recursive registrations using multiple initial reference images. In addition, this embodiment differs from the conventional method in that the adjustment amount calculation unit 234 calculates adjustment amounts for adjusting the relative positions of multiple strips included in an image set based on such multiple pieces of relative position information.

[0214] Any method may be used to calculate the adjustment amount for adjusting the relative positions of multiple images included in an image set. Some non-limiting examples are described below. In these examples, a method for calculating the adjustment amount for adjusting the relative positions in the lateral direction (x and y directions) is described, but it will be understood by those skilled in the art that the adjustment amount for adjusting the relative positions in the axial direction (z direction) can also be calculated using a similar method.

[0215] An example of a process for calculating an adjustment amount for adjusting the relative position in the lateral direction will be described with reference to Figures 9A to 9D and 10. In this example, the initial reference image selection method of Figure 6 (initial reference image selection based on the time axis of optical scanning) is applied, but similar processes can also be executed when other initial reference image selection methods are applied.

[0216] The time axis t in the upper part and the time axis in the lower part of FIG. 9A are both time axes of the optical scanning (Lissajous scanning) performed in this example. Symbol F indicates an image set (plurality of strips) acquired using the Lissajous scanning. In this example, three intervals T1, T2, and T3 are set in chronological order with respect to the time axis t of the Lissajous scanning. Symbol R1 denotes an image (strip) selected by the image selection unit 225 from the image group (strip group) corresponding to interval T1, symbol R2 denotes an image (strip) selected by the image selection unit 225 from the image group (strip group) corresponding to interval T2, and symbol R3 denotes an image (strip) selected by the image selection unit 225 from the image group (strip group) corresponding to interval T3. The three strips R1, R2, and R3 are each used as initial reference images (initial reference strips) for recursive registration, which is performed three times. The symbol t1 denotes the time corresponding to the first initial reference strip R1, the symbol t2 denotes the time corresponding to the second initial reference strip R2, and the symbol t3 denotes the time corresponding to the third initial reference strip R3.

[0217] The coordinate system in the lower part of FIG. 9A is spanned by the time axis t of the Lissajous scan and a coordinate axis ΔX indicating the lateral positional deviation (shift amount). The lateral shift amount ΔX in FIG. 9A may be a vector amount or a scalar amount. The vector amount ΔX includes, for example, a shift amount in the x direction and a shift amount in the y direction: ΔX=(Δx, Δy). The scalar amount ΔX may be, for example, a shift amount in the x direction (ΔX=Δx), a shift amount in the y direction (ΔX=Δy), or a shift amount in any direction on the xy plane. Below, we will explain the case where ΔX is a vector amount ΔX=(Δx, Δy), but similar processing can be performed when ΔX is another vector amount or scalar amount.

[0218] 9A shows three graphs G1, G2, and G3 expressed in such a coordinate system (t, ΔX). The first graph G1 represents the x- and y-direction shift amounts (x- and y-direction shift amount data set, relative position information) of each strip obtained by the first recursive registration performed using the first initial reference strip R1 as a starting point. The second graph G2 represents the x- and y-direction shift amounts (x- and y-direction shift amount data set, relative position information) of each strip obtained by the second recursive registration performed using the second initial reference strip R2 as a starting point. The third graph G3 represents the x- and y-direction shift amounts (x- and y-direction shift amount data set, relative position information) of each strip obtained by the third recursive registration performed using the third initial reference strip R3 as a starting point.

[0219] In this example, the data described above is input from the shift amount recording unit 233 to the adjustment amount calculation unit 234. The adjustment amount calculation unit 234 in this example first executes a process of aligning the x and y direction shift amounts ΔX of the three graphs G1 to G3 in order to make the x and y direction shift amounts indicated by the three graphs G1 to G3 comparable.

[0220] For example, the adjustment amount calculation unit 234 aligns the xy direction shift amounts ΔX of the three graphs G1 to G3 using the following formula: ΔX′ i (R(n+1)) =ΔX i(R(n+1)) +ΔX´ R(n+1) (Rn) where ΔX denotes the xy direction shift amount vector ΔX=(Δx, Δy) before alignment; ΔX′ denotes the xy direction shift amount vector ΔX′=(Δx′, Δy′) after alignment; i denotes the index of the strip based on the individual identifier described above; and Rn denotes the strip selected by the image selection unit 225 from the group of strips corresponding to the interval Tn (n=1, 2, 3) set on the time axis t of the Lissajous scan.

[0221] 9B shows a process for aligning the first graph G1 and the second graph G2. In this alignment process, the adjustment amount calculation unit 234 first calculates the difference (difference vector, vector difference) between the x- and y-direction shift amount of the second graph G2 at time t2 and the x- and y-direction shift amount of the first graph G1 at time t2. Furthermore, the adjustment amount calculation unit 234 moves the second graph G2 in the ΔX-axis direction by the calculated difference. This alignment process translates the second graph G2 in the ΔX-axis direction so as to cancel the difference between the first graph G1 and the second graph G2 at time t2. The second graph G2 aligned with the first graph G1 is denoted by the symbol G2'.

[0222] Similarly, FIG. 9C illustrates a process for aligning the second graph G2′, which is aligned with the first graph G1, with the third graph G3. In this alignment process, the adjustment amount calculation unit 234 first calculates the difference (difference vector, vector difference) between the x- and y-direction shift amount of the second graph G2′ at time t3 and the x- and y-direction shift amount of the third graph G3 at time t3. Furthermore, the adjustment amount calculation unit 234 shifts the third graph G3 in the ΔX-axis direction by the calculated difference. This alignment process translates the third graph G3 in the ΔX-axis direction so as to cancel the difference between the second graph G2′ and the third graph G3 at time t3. The third graph G3 aligned with the second graph G2′ is indicated by the symbol G3′.

[0223] As a result, three graphs G1, G2', and G3' shown in FIG. 9D are obtained. Note that the alignment process for making multiple graphs comparable is not limited to the above example and may be any process. For example, all the other graphs may be aligned with one of the multiple graphs. Also, an algorithm may be configured to compare multiple graphs without performing alignment process.

[0224] Furthermore, the adjustment amount calculation unit 234 in this example may be configured to select, for any i-th strip, the initial reference strip that minimizes the sum of the geometric distances between the x- and y-direction shift amounts of this i-th strip and the x- and y-direction shift amounts of other i-th strips as the optimal initial reference strip, and to specify the x- and y-direction shift amounts obtained by recursive registration based on this optimal initial reference strip as the x- and y-direction shift amounts of the i-th strip.

[0225] In other words, the adjustment amount calculation unit 234 in this example may select an xy direction shift amount vector from among the three xy direction shift amount vectors calculated for the i-th strip such that the sum of the geometric distances between the xy direction shift amount vector and the other two xy direction shift amount vectors is the smallest, and adopt the selected xy direction shift amount vector as the optimal xy direction shift amount vector for this i-th strip.

[0226] Such an operation can be expressed, for example, as the following equation (see FIG. 10): ΔX' i =ΔX´ i (R´i) ;R´ i =argmin Rj [Σ Rj≠Rk {ΔX´ i (Rk) -ΔX´ i (Rj)} 2 ]. Here, the sum Σ is calculated over Rk other than Rj (Rj ≠ Rk), and the minimum point set argmin (argument of the minimum) is calculated over Rj. Also, ΔX' i (Rk) -ΔX´i (Rj) corresponds to a vector indicating the amount of misalignment in the x and y directions between the two ith strips.

[0227] While the calculation of the adjustment amount in the lateral direction (x and y directions) has been described here, the adjustment amount in the axial direction (z direction) can also be calculated in the same manner. Also, while the optimal shift amount is calculated independently for each strip here, it is also possible to determine the optimal shift amount using information about strips that are located nearby in time or space. For example, in the calculation for the i-th strip, it is possible to use information about the i-1-th strip and information about the i+1-th strip.

[0228] In this way, the process of this example performs recursive registration multiple times using multiple initial reference images, and selects the optimal shift amount for each strip from the multiple shift amount data obtained by these recursive registrations. This differs from conventional methods that perform recursive registration once using a single initial reference image. Furthermore, the alignment process and optimal initial reference image selection process in this example are novel technologies unique to this embodiment that are not found in conventional methods.

[0229] <Image data correction unit 235> The image data corrector 235 is configured to adjust the relative positions of the images included in the image set based at least on the adjustment amount calculated by the adjustment amount calculator 234 .

[0230] For example, the image data correction unit 235 in the embodiment shown in Figure 4C adjusts the relative positions in the lateral direction (x and y directions) of multiple strips included in an image set based on the lateral adjustment amount (x and y direction shift amount) of each strip calculated by the adjustment amount calculation unit 234, and also adjusts the relative positions in the axial direction (z direction) of multiple sub-volumes corresponding to these multiple strips based on the z direction shift amount of each sub-volume calculated by the z shift amount calculation unit 232.

[0231] In addition, in an aspect in which the adjustment amount calculation unit 234 is configured to calculate the adjustment amount in the axial direction, the image data correction unit 235 can adjust the relative positions in the lateral direction of multiple strips included in the image set based on the lateral adjustment amount of each strip calculated by the adjustment amount calculation unit 234, and can also adjust the relative positions in the axial direction of multiple sub-volumes corresponding to these multiple strips based on the axial adjustment amount of each sub-volume calculated by the adjustment amount calculation unit 234.

[0232] The following describes adjustment of the relative positions of multiple subvolumes in the axial direction based on the z-direction shift amounts calculated by the z-shift amount calculation unit 232. Similar to the registration unit 2317 that performs registration based on the xy-direction shift amounts, the image data correction unit 235 performs registration between the reference subvolume and the target subvolume so as to cancel out the corresponding z-direction shift amounts obtained by the z-shift amount calculation unit 232.

[0233] In some example embodiments, the image data corrector 235 may be controlled to perform registration between the reference sub-volume and the target sub-volume only if the corresponding z-shift exceeds a threshold.

[0234] The image set obtained as a result of the position adjustment by the image data corrector 235 is a group of subvolumes that have been registered (adjusted relative positions) in all of the x, y, and z directions, i.e., a group of subvolumes that have been three-dimensionally registered. The group of subvolumes before applying such registration is a volume acquired by Lissajous scanning. The data processor 230 recombines the group of subvolumes obtained by three-dimensional registration to obtain a volume to which three-dimensional registration has been applied.

[0235] The 3D registered dataset can then be used for any processing (image processing, post-processing, analysis, evaluation, visualization, rendering, etc.) This allows the desired processing to be performed using a 3D dataset that has been corrected for motion artifacts with higher accuracy and precision than ever before.

[0236] In addition, the z-shift amount calculation unit 232 and the image data correction unit 235 may be configured to perform registration equivalent to "axial motion correction (rough axial motion correction and / or fine axial motion correction)" (pages 1789 to 1790) of Non-Patent Document 1 in order to remove motion artifacts in the z direction.

[0237] <User Interface 240> The user interface 240 includes a display unit 241 and an operation unit 242. The display unit 241 includes the display device 3. The operation unit 242 includes various operation devices and input devices. The user interface 240 may include a device that combines a display function and an operation function, such as a touch panel. It is also possible to construct an embodiment that does not include at least a part of the user interface 240. For example, the display device may be an external device connected to the ophthalmic examination apparatus 1.

[0238] <Data Receiving Unit 250> The data receiving unit 250 acquires data from an external device. This external device may be, for example, a computer, a storage device, a recording medium, or an information system (for example, a hospital information system, an electronic medical record system, or an image archiving system). The external device may be, for example, a device directly connected to the ophthalmic examination apparatus 1, a device connected to the ophthalmic examination apparatus 1 via a local area network (LAN), or a device connected to the ophthalmic examination apparatus 1 via a wide area network (WAN).

[0239] The data receiving unit 250 may include, for example, a communication interface, a drive device, etc. The data receiving unit 250 may also include a scanner for reading data recorded on paper sheets, etc. The data receiving unit 250 may also include an input device for a user to input data. The input device may be a keyboard, a pen tablet, etc.

[0240] <Operation> Some examples of the operation of the ophthalmic examination apparatus 1 will now be described.

[0241] <First operation example> In the first operation example, the operation shown in Fig. 11 will be described. In this operation example, multiple initial reference images are selected from an image set generated based on redundant data collection, and these initial reference images are used to perform relative position adjustment of the image set by multiple registrations. In this operation example, the redundant data collection is a Lissajous scan, the image set is a series of strips (multiple strips), and the initial reference image is an initial reference strip.

[0242] Prior to step S1, the same preparatory operations as in the conventional method are performed, such as inputting the patient ID, setting the scanning mode (specifying Lissajous scan), presenting the fixation target, alignment, focus adjustment, and OCT optical path length adjustment.

[0243] (S1: Collect volume using Lissajous scan) Upon receiving a predetermined scan start trigger signal, the scan control unit 2111 starts applying an OCT scan (Lissajous scan) to the subject's eye E (fundus Ef). The scan start trigger signal is generated, for example, in response to the completion of predetermined preparatory operations (alignment, focus adjustment, OCT optical path length adjustment, etc.) or in response to a scan start instruction operation being performed using the operation unit 242. The scan control unit 2111 applies a Lissajous scan to the fundus Ef by controlling the optical scanner 44, the OCT unit 100, etc. based on a scan protocol 2121 (a protocol corresponding to the Lissajous scan). A data set (volume) collected by the Lissajous scan is sent to the image data construction unit 220.

[0244] (S2: Divide into a series of subvolumes) The image data constructing unit 220 divides the volume acquired in step S1 into a series of sub-volumes, each of which is considered to be unaffected by the movement of the subject's eye E.

[0245] (S3: Build a series of strips) The image data constructing unit 220 constructs a front projection image (strip) of each sub-volume obtained in step S2. This constructs a series of strips (image sets) from the series of sub-volumes generated in step S2. The series of strips is sent to the data processing unit 230.

[0246] (S4: Select multiple initial reference strips) The image selector 225 selects a number of initial reference strips from the set of strips constructed in step S3, using any of the initial reference image selection methods described above.

[0247] (S5: Perform multiple recursive registrations) The image data processing unit 231 applies multiple recursive registrations to the series of strips, each of which is based on the multiple initial reference strips selected in step S4. One set of x and y direction shift amount data (relative position information) is generated for the series of strips by the recursive registration based on one initial reference strip. By performing the recursive registration based on each of the multiple initial reference strips, multiple sets of x and y direction shift amount data (multiple sets of relative position information) are generated for the series of strips.

[0248] (S6: Calculate the relative position adjustment amount of a series of strips) The image data processing unit 231 calculates the adjustment amount for adjusting the relative positions of the series of strips (motion artifact correction, particularly motion artifact correction in the x and y directions) based on the plurality of pieces of relative position information generated in step S5. This process is performed using any of the relative position adjustment amount calculation methods described above.

[0249] (S7: Perform relative position adjustment of a series of strips) The image data processing unit 231 performs a series of relative position adjustments of the strips based on the adjustment amounts calculated in step S6. In this step, relative position adjustments in the lateral direction (x and y directions) are performed. In this step, processing similar to "fine lateral motion correction" in Non-Patent Document 1 may also be performed.

[0250] (S8: Perform relative position adjustment of a series of subvolumes) The image data processing unit 231 performs relative position adjustment of the series of sub-volumes generated in step S2. This relative position adjustment includes relative position adjustment in the lateral direction (xy direction) and relative position adjustment in the axial direction (z direction). The result of the relative position adjustment of the series of strips performed in step S7 is used for the relative position adjustment in the lateral direction. Furthermore, the relative position adjustment in the axial direction (z direction) may be processing equivalent to "axial motion correction (rough axial motion correction and / or fine axial motion correction)" in Non-Patent Document 1.

[0251] This is the end of the processing according to this operation example (end).

[0252] <Second operation example> In the second operation example, the operation shown in Fig. 12 will be described. This operation example provides one example of the processing of steps S4 to S7 of the first operation example. The processing of step S11 of this operation example is executed following the processing of step S3 of the first operation example, and the processing of step S8 of the first operation example is executed after the processing of step S20 of this operation example.

[0253] In this operation example, an example of dividing an image set obtained by redundant data collection in time is described, but processing similar to that of this operation example can also be performed when dividing the image set in space or when other division methods are applied.

[0254] (S11: Divide the series of strips into K groups of strips) For example, in the same manner as the process described with reference to FIG. 6, the image selection unit 225 divides the series of strips constructed in step S3 into K strip groups (K is an integer equal to or greater than 2).

[0255] (S12: Select an initial reference strip from each strip group) Next, the image selection unit 225 selects an initial reference strip from each of the K strip groups generated in step S11. In this example, one initial reference strip is selected from each strip group, but two or more initial reference strips may be selected from any strip group, or no initial reference strip may be selected from any strip group.

[0256] (S13: Start registration (k=1)) The image data processing unit 231 starts a plurality of registrations using the plurality of initial reference strips selected in step S12 as references, respectively.

[0257] The index of the K initial reference strips is represented by k (k=1, 2, . . . , K). The index k is assigned to the multiple initial reference strips in any order. For example, the index k may be assigned in order according to the time axis (see FIGS. 9A to 9D), in order according to the size of the strip, or in order according to other information.

[0258] In this operation example, a case will be described in which K registrations based on K initial reference strips are sequentially performed in an order according to index k=1 to K (see steps S14 to S16 below). However, the order in which the K registrations based on the K initial reference strips are performed is not limited to the order according to index k, and may be any order.

[0259] In some example embodiments, at least two of the K registrations based on the K initial reference strips may be performed at least partially in parallel, e.g., for a first and a second of the K registrations, the first and second registrations may be performed in parallel such that at least a portion of the time taken for the first registration and at least a portion of the time taken for the second registration overlap each other in time.

[0260] (S14: k = K? If the index k of the initial reference strip to be processed is not equal to K (S14: No), that is, if the index k of the initial reference strip to be processed is less than K (k < K), the process proceeds to step S15. On the other hand, if the index k of the initial reference strip to be processed is equal to K (S14: Yes), the process proceeds to step S17.

[0261] (S15: Execute recursive registration using the k-th initial reference strip) When it is determined as "No" in step 14, the image data processing unit 231 (mask image generation unit 2311, range adjustment unit 2312, composite image generation unit 2313, cross-correlation function calculation unit 2314, correlation coefficient calculation unit 2315, xy shift amount calculation unit 2316, registration unit 2317, and merge processing unit 2318) executes recursive registration based on the k-th initial reference strip.

[0262] In each stage of the iterative process in the recursive registration based on the k-th initial reference strip, the xy-direction shift amount data calculated by the xy shift amount calculation unit 2316 is sent to and recorded in the shift amount recording unit 233 together with the additional information. This additional information includes, for example, an identifier corresponding to the k-th initial reference strip (the aforementioned common identifier) and an identifier of the xy-direction shift amount data (the aforementioned individual identifier). This individual identifier may be, for example, an identifier of the target strip in the registration at the stage where the xy-direction shift amount data is acquired (or an identifier compatible with the identifier of the target strip). [[ID=十六]] [[ID=十七]]

[0263] [[ID=十八]] (S16: k = k + 1) [[ID=二十]] [[ID=二十一]]When the recursive registration (step S15) based on the k-th initial reference strip is completed, the process proceeds to the recursive registration based on the (k + 1)-th initial reference strip. [[ID=二十二]] [[ID=二十三]]

[0264] [[ID=二十四]] By repeatedly executing the processing of steps S14 to S16 until the result of step S14 is "Yes," multiple recursive registrations are performed, each based on the K initial reference strips selected in step S12, and K sets of xy direction shift amount data (K pieces of relative position information) are obtained and recorded in the shift amount recording unit 233.

[0265] Each element (each xy-direction shift amount data) of each xy-direction shift amount data set is assigned a common identifier corresponding to the initial reference strip in the recursive registration that generated the xy-direction shift amount data set, and an individual identifier for the element. Also, each of the K xy-direction shift amount data sets (K pieces of relative position information) includes xy-direction shift amount data for each strip in the series of strips. In other words, the shift amount recording unit 233 records K pieces of xy-direction shift amount data for each strip constructed in step S3.

[0266] (S17: Perform relative alignment of K xy direction shift amount data sets) Next, in the same manner as the processing described with reference to, for example, FIGS. 9A to 9D, the adjustment amount calculation unit 234 performs relative alignment of the K xy direction shift amount data sets (relative position information) generated by the multiple recursive registrations of steps S14 to S16.

[0267] (S18: Determine the optimal x and y direction shift amount for each strip) Next, the adjustment amount calculation unit 234 determines the optimum x and y direction shift amounts for each strip in the series of strips based on the K x and y direction shift amount data sets that have been aligned in step S17.

[0268] For example, for each strip in a series of strips, the adjustment amount calculation unit 234 first extracts x and y direction shift amount data corresponding to that strip from each of the K x and y direction shift amount data sets. Next, the adjustment amount calculation unit 234 selects the optimal x and y direction shift amount for that strip from the extracted K x and y direction shift amount data in the same manner as the process described with reference to Fig. 10. This allows optimal x and y direction shift amounts to be obtained for all of the series of strips. Note that the method of selecting the optimal one from the K x and y direction shift amount data is not limited to this.

[0269] Furthermore, the process of determining the optimal x and y direction shift amounts for each strip is not limited to the process of selecting the optimal one from K pieces of x and y direction shift amount data. For example, a statistic obtained by applying a predetermined statistical calculation to K pieces of x and y direction shift amount data, or an amount calculated from this statistic, may be used as the optimal x and y direction shift amount for that strip.

[0270] Alternatively, to determine the optimal x and y direction shift amounts for a strip, other data may be used in addition to or instead of the K x and y direction shift amount data corresponding to the strip. In some exemplary embodiments, x and y direction shift amount data corresponding to one or more other strips (neighboring strips) located close to the strip in time or space can be used. For example, the optimal x and y direction shift amounts for the strip can be determined by applying a predetermined operation (such as a statistical operation or curve fitting) to the x and y direction shift amount data (of the strip and) the x and y direction shift amount data of the neighboring strips.

[0271] (S19: Adjust the relative positions of a series of strips) The image data correction unit 235 adjusts the relative positions of the series of strips based on the optimal x and y direction shift amounts determined for each of the series of strips in step S18. This relative position adjustment is a relative position adjustment in the lateral direction (x and y directions) and is performed by moving each strip by a vector equivalent to its optimal x and y direction shift amount. The relative position adjustment in this step is a process equivalent to the "rough lateral motion correction" in Non-Patent Document 1, performed based on the optimal x and y direction shift amounts determined in step S18.

[0272] (S20: Perform precise relative position adjustment of a series of strips) The image data correction unit 235 can further perform a more precise adjustment of the relative position in the lateral direction than the adjustment of the relative position in the lateral direction in step S19. The precise adjustment of the relative position in this step may be a process equivalent to the “fine lateral motion correction” in Non-Patent Document 1.

[0273] When step S20 is completed, step S8 in FIG. 11 is executed, and the processing according to this operation example ends (END).

[0274] <Third operation example> In the third operation example, an example of recursive registration executed in step S5 of FIG. 11 and step S15 of FIG. 12 will be described with reference to FIGS. 13A and 13. FIG.

[0275] (S31: Set the reference strip and target strip) In recursive registration using a given initial reference strip (one of the multiple initial reference strips selected in step S4 of FIG. 11 ), the data processing unit 230 first orders the series of strips constructed in step S3 of FIG. 11 according to size (such as area). In this example, the number of strips in the series is assumed to be N (N is an integer equal to or greater than 2). In this example, these N strips will be referred to as the first strip, the second strip, . . . , the Nth strip, according to the order specified by the ordering. In this example, any strip from the first to Nth strips may be referred to as the nth strip (n=1, 2, . . . , N). Thus, in this example, the first strip is the largest strip, the second strip is the second largest strip, the nth strip is the nth largest strip, and the Nth strip is the smallest strip.

[0276] The given initial reference strip is any one of the first to Nth strips. In this example, the given initial reference strip is the first strip, and the initial target strip is the second strip. If the given initial reference strip is not the first strip, the given initial reference strip is set as the initial reference strip, and the first strip is set as the initial target strip. Those skilled in the art will understand that recursive registration similar to this example can also be performed in this case.

[0277] The reference strip and the target strip correspond to the arbitrary shaped images f(r) and g(r) in Appendix A of Non-Patent Document 1, respectively.

[0278] (S32: Set the reference mask image and the target mask image) The mask image generation unit 2311 generates a mask image corresponding to the reference strip (reference mask image) and a mask image corresponding to the target strip (target mask image).

[0279] In this stage (first registration in the recursive registration), the mask image generation unit 2311 generates a mask image corresponding to the first strip, which is the initial reference strip, and a mask image corresponding to the second strip. The data processing unit 230 sets the first mask image corresponding to the first strip as the initial reference mask image, and sets the second mask image corresponding to the second strip as the initial target mask image.

[0280] The reference mask image and the target mask image are rectangular shaped binary image masks m in Appendix A of Non-Patent Document 1, respectively. f (r) and m g Equivalent to (r).

[0281] (S33: Normalize the reference strip and the target strip) The range adjustment unit 2312 normalizes the reference strip and the target strip set in step S31 in the manner described above. The strips are normalized according to the range of pixel values ​​of the mask image. In this stage (first registration in the recursive registration), the range adjustment unit 2312 applies normalization to the first strip (reference strip) and the second strip (target strip). Note that the processing performed by the range adjustment unit 2312 is not limited to normalization, and may be any of the range adjustment processing described above or a processing similar thereto.

[0282] In this example, the pixel value range of each mask image is assumed to be in the closed interval [0, 1]. In particular, each mask image in this example is assumed to be a binary image in which the pixel values ​​within the region corresponding to the domain of the corresponding strip are set to 1 and the other pixel values ​​are set to 0.

[0283] The range adjustment unit 2312 normalizes the strip, for example, by dividing the value of each pixel in the strip by the maximum pixel value in the strip, or by dividing the value of each pixel in the strip by the maximum value of the range of pixel values ​​(tone range) of the strip.

[0284] Furthermore, the data processing unit 230 (synthetic image generation unit 2313) embeds the strip with the normalized pixel value range into an image of the same size and shape as the corresponding mask image. The embedded image of the normalized first strip f(r) is similar to the rectangular shaped image f'(r) in Appendix A of Non-Patent Document 1, but the absolute value (|f(r)|) of the value range within the image area of ​​the first strip f(r) is normalized to 1 or less. This embedded image is also represented as f'(r).

[0285] Similarly, the embedded image of the normalized second strip g(r) is an image similar to the rectangular image g'(r), but the absolute value of the range within the image area of ​​the second strip g(r) (|g(r)|) is normalized to be less than or equal to 1. This embedded image is also denoted as g'(r).

[0286] (S34: Generate a reference composite image and a target composite image) The composite image generator 2313 generates a composite image of the embedded image of the normalized strip and the corresponding mask image.

[0287] In this stage (first registration in the recursive registration), the composite image generation unit 2313 generates a composite image of the embedded image of the normalized first strip (initial reference strip) and the first mask image (reference mask image), and also generates a composite image of the embedded image of the normalized second strip (initial target strip) and the second mask image (target mask image). The former composite image is called the reference composite image, and the latter is called the target composite image.

[0288] Here, the reference composite image and the target composite image are the combination of two rectangle shaped images f'(r)m in Appendix A of Non-Patent Document 1, respectively. f (r) and g´(r)m g (r), where, as mentioned above, the absolute value of the range of the first strip f(r) in the image area (|f(r)|) is less than or equal to 1, and the absolute value of the range of the second strip g(r) in the image area (|g(r)|) is less than or equal to 1.

[0289] Image 311 in Figure 14A is an example of an embedded image f'(r) of the normalized first strip f(r), and image 321 is an example of a first mask image. Combining these two images 311 and 321 produces a reference composite image f'(r)m f (r) is obtained.

[0290] Similarly, image 312 in Figure 14B is an example of an embedded image g'(r) of the normalized second strip g(r), and image 322 is an example of a second mask image. Combining these two images 312 and 322 produces a reference composite image g'(r)m g (r) is obtained.

[0291] (S35: Calculate multiple cross-correlation functions) The cross-correlation function calculation unit 2314 calculates a plurality of cross-correlation functions based on the reference composite image and the target composite image generated in step S34. In this example, the cross-correlation function calculation unit 2314 calculates six cross-correlation functions included in equation (33) of Non-Patent Document 1 based on the reference composite image and the target composite image generated in step S34.

[0292] (S36: Calculate the correlation coefficient) The correlation coefficient calculation unit 2315 calculates a correlation coefficient based on the multiple cross-correlation functions calculated in step S35. In this example, the correlation coefficient calculation unit 2315 calculates the correlation coefficient (ρ(r')) from the multiple cross-correlation functions calculated in step S35 according to equation (33) in Non-Patent Document 1.

[0293] (S37: Record the correlation coefficient) The shift amount recording unit 233 can record the correlation coefficient (ρ(r')) calculated in step S36 as additional information of the x and y direction shift amounts calculated in the next step S38. As described above, this additional information includes identifiers (common identifier, individual identifier) ​​of the x and y direction shift amount data.

[0294] (S38: Calculate the shift amount in the x and y directions between the reference strip and the target strip) The xy shift amount calculation unit 2316 calculates the amount of shift in the xy directions between the first strip (initial reference strip) f(r) and the second strip (initial target strip) g(r) based on the correlation coefficient calculated in step S36. In this example, the relative amount of shift (Δx, Δy) in the xy directions between the reference strip f(r) and the target strip g(r) is found by detecting the peak of the correlation coefficient ρ(r') calculated in step S36.

[0295] (S39: Record the shift amount in the x and y directions) The xy direction shift amounts (xy direction shift amount data) calculated in step S38 for the pair of the reference strip and the target strip are recorded by the shift amount recording unit 233 together with accompanying information.

[0296] (S40: Adjust the relative position between the reference strip and the target strip) The registration unit 2317 adjusts the relative position between the reference strip and the target strip based on the shift amounts in the x and y directions calculated in step S38. This relative position adjustment is a relative position adjustment in the lateral direction (x and y directions) (rough lateral motion correction). In this stage (first registration in the recursive registration), the relative position adjustment between the first strip f(r) and the second strip g(r) is performed.

[0297] Furthermore, the registration unit 2317 may apply any xy registration to the reference strip and the target strip, such as the "fine lateral motion correction" described above.

[0298] (S41: Construct a merged image of the reference strip and the target strip) The merge processing unit 2318 constructs a merged image of the reference strip and the target strip whose relative positions have been adjusted in step S40. Image 330 in Figure 14C shows an example of a merged image of the first strip f(r) and the second strip g(r).

[0299] Here, an example of implementing the processes in steps S35 to S41 will be described. First, f'(r)m f (r), g´(r)m g (r), m f (r), m g (r), (f´(r)m f (r)) 2 , and (g´(r)m g (r)) 2 By setting the imaginary part of each of the above to 0, we get only the real part.

[0300] Next, f´(r)m f (r), g´(r)m g (r), m f (r), m g (r), (f´(r)m f (r)) 2, and (g´(r)m g (r)) 2 A fast Fourier transform (FFT) is applied to each real part of

[0301] Next, based on the group of functions obtained by these fast Fourier transforms, the six cross-correlation functions shown in step S35 of FIG. 13A are derived.

[0302] An inverse fast Fourier transform (IFFT) is then applied to each of the six derived cross-correlation functions.

[0303] Next, the correlation coefficient ρ(r′) of equation (33) in Non-Patent Document 1 is calculated.

[0304] Then, the relative shift amount (Δx, Δy) between the reference strip f(r) and the target strip g(r) is calculated by identifying the peak position of the correlation coefficient ρ(r'), and registration and merging processes between the reference strip f(r) and the target strip g(r) are performed based on this shift amount.

[0305] By applying this series of processes to the N strips in turn, we can obtain all merged images of the N strips. In addition, the xy direction shift amount (Δx, Δy) calculated for each strip pair can be saved together with the time information (i.e., associated with the time information).

[0306] (S42: Have you processed all the strips?) A series of processes from steps S31 to S41 is executed for all of the series of strips (N strips) obtained in step S3, in the order described above.

[0307] When N is 3 or more, if a merged image of the first strip and the second strip is created in step S41 (S42: No), the process returns to step S31.

[0308] In the second step S31 (step S31 of the second registration in the recursive registration), a merged image of the first strip and the second strip is set as a new reference strip, and the third strip is set as a new target strip. By performing the processes of steps S32 to S41 based on the new reference strip and the new target strip, xy direction shift amount data and correlation coefficients between the new reference strip and the new target strip are generated and recorded, and a merged image of the new reference strip and the new target strip is obtained. This new merged image is a merged image of the first to third strips.

[0309] By applying this series of processes to the N strips in turn, merged images of all the N strips are obtained.

[0310] When the processing of all the series of strips obtained in step S3 is completed (S42: Yes), the process proceeds to recursive registration based on the next initial reference strip (step S16 in FIG. 12).

[0311] <Modification> A modified example of the ophthalmic examination apparatus 1 according to the above embodiment will be described. In the above exemplary embodiment, the image size is referenced to select multiple initial reference images from an image set constructed from a data set acquired by redundant data collection, and the initial reference strip with the largest area is selected from each of multiple subsets of multiple strips (series of strips) obtained using Lissajous scanning, and multiple registrations are performed using correlation coefficients between the strips.

[0312] When using correlation coefficients in this way, motion artifact correction can be performed by discarding data (e.g., A-lines, cycles, strips) with low correlation coefficients with the initial reference image. The amount of data discarded due to low correlation depends on the selection of the initial reference image. That is, recursive registration starting from one initial reference image may discard a relatively large amount of data, while recursive registration starting from a different initial reference image may discard a relatively small amount of data. In particular, using a large initial reference image does not necessarily reduce the amount of discarded data.

[0313] This modified example is based on the findings of the inventors and selects an initial reference image so as to reduce the amount of discarded data. More specifically, this modified example evaluates the amount of discarded data (e.g., determines whether the amount of discarded data is less than a predetermined threshold) for images in descending order of size, thereby selecting an image that is as large as possible and has as little discarded data as possible as the initial reference image.

[0314] The processing according to this modification includes, for example, the following multiple steps. First, in this example, a volume is acquired by Lissajous scanning, divided into a series of sub-volumes, and a series of strips is constructed, similar to the operational example of Fig. 11. Next, in this example, the series of strips is divided into a plurality of strip groups, similar to the operational example of Fig. 12.

[0315] Next, in this example, the multiple strips in each strip group are ordered in descending order of size (area). These multiple strips will be called the first strip, second strip, etc. in this ordering order.

[0316] Next, this example performs recursive registration ("rough lateral motion correction" in Non-Patent Document 1) using the first strip as a reference, and calculates the usage rate of the A-line in the image obtained thereby.

[0317] The A-line usage rate may be, for example, the quotient value obtained by dividing the number of A-lines used to construct the image (in other words, the number of A-lines included in the image, or further in other words, the number of A-lines not discarded during the recursive registration process) by the number of A-lines in the original volume. This A-line usage rate is equal to the quotient value obtained by dividing the number of A-lines having a correlation coefficient value equal to or greater than a predetermined threshold by the total number of A-lines. Therefore, when correlation coefficients are recorded as in the operational examples of FIGS. 13A and 13B , the A-line usage rate can be calculated based on multiple correlation coefficients recorded during recursive registration. Note that the index used for evaluation is not limited to the A-line usage rate, and may be a value equivalent to the A-line usage rate (e.g., the A-line non-use rate), the usage rate or non-use rate of cycles, the usage rate or non-use rate of strips, etc.

[0318] In this example, if the A-line usage rate of the first strip is equal to or greater than a predetermined threshold, the first strip is designated as the initial reference strip selected from this group of strips. On the other hand, if the A-line usage rate of the first strip is less than the predetermined threshold, the example moves on to processing the second strip, which is the next largest in size.

[0319] Furthermore, in this example, the same processing as that for the first strip is applied to the second strip. In this way, by evaluating the A-line usage rate in the strips in order of size, it becomes possible to select an initial reference image taking both size and A-line usage rate into consideration.

[0320] Alternatively, the A-line usage rates of at least some of the strips in the strip group may be calculated, and the strip with the largest A-line usage rate may be designated as the initial reference strip. At this time, one or more strips having a size equal to or larger than a predetermined threshold may be extracted from the strip group, and the strip with the largest A-line usage rate among the extracted one or more strips may be designated as the initial reference strip.

[0321] <Effects> Some effects of the ophthalmic examination apparatus 1 according to this embodiment will be described.

[0322] The ophthalmic examination apparatus 1 is a scanning imaging apparatus that performs imaging of a sample using optical scanning, and includes as its functional elements a data set acquisition unit, an image set generation unit, an image selection unit, and an image position adjustment unit.

[0323] The dataset acquisition unit is configured to acquire a dataset by redundantly collecting data of the sample by optical scanning. In this embodiment, the dataset acquisition unit may include an optical scanner 44, an OCT unit 100, and a control unit 210, and the optical scanning may be a Lissajous scan, but these are non-limiting examples.

[0324] The image set generator is configured to generate an image set based on the dataset acquired by the dataset acquirer. The generated image set is a plurality of images having positional redundancy. In this embodiment, the image set generator may include an image data constructor 220, and the image set may be a plurality of strips (a series of strips) or a plurality of sub-volumes (a series of sub-volumes), but these are non-limiting examples.

[0325] The image selector selects a plurality of images from the image set generated by the image set generator. The selected plurality of images are used as initial reference images. In this embodiment, the image selector may include the image selector 225, and the selected plurality of images from the image set may be the plurality of initial reference strips, but these are non-limiting examples.

[0326] The image position adjustment amount is configured to adjust the relative positions of the image set by applying a plurality of registrations to the image set, each of which is based on a plurality of images selected from the image set by the image selection unit. In this embodiment, the image position adjustment unit may include the image data processing unit 231, and the registration may be recursive registration using a correlation coefficient, but these are non-limiting examples.

[0327] As mentioned above, in conventional techniques, registration is performed using only a single initial reference image, which can result in the selection of an initial reference image having a negative impact on the quality of the final image. However, according to this embodiment, multiple registrations are performed using multiple initial reference images, which solves this problem, reduces errors in motion artifact correction, and enables improved motion artifact correction.

[0328] The ophthalmic examination apparatus 1 according to this embodiment further has the following non-limiting features (non-limiting functions, non-limiting configurations, etc.). Those skilled in the art will understand that these non-limiting features contribute to further improvement of motion artifact correction and provide actions and effects according to each non-limiting feature. Note that the features of the ophthalmic examination apparatus 1 according to this embodiment are not limited to the following.

[0329] The image position adjustment unit may be configured to generate a plurality of pieces of relative position information for the image set generated by the image set generation unit by applying a plurality of registrations to the image set. The relative position information is information that represents a relative positional relationship between a plurality of images included in the image set. Furthermore, the image position adjustment unit may be configured to perform relative position adjustment for the image set based on the generated plurality of pieces of relative position information.

[0330] The image position adjustment unit may be configured to generate relative position information for the image set by applying recursive registration to the image set using one of the multiple images selected by the image selection unit as an initial reference image in each of the multiple registrations.

[0331] The image position adjustment unit may be configured to determine an adjustment amount for adjusting the relative positions of the image set based on the plurality of pieces of relative position information generated by the plurality of registrations. The adjustment amount may be, for example, a plausible positional deviation amount (optimal position correction amount) for each image included in the image set.

[0332] The image selector may be configured to select a plurality of images from the image set generated by the image set generator based on a time axis of the optical scanning performed by the data set acquirer.

[0333] The image selector may be configured to divide the image set generated by the image set generator into a plurality of subsets corresponding to a plurality of time periods of the optical scanning performed by the dataset acquirer, and further configured to select a plurality of images from the image set by selecting an image from each of the obtained subsets.

[0334] The image selector may be configured to select a plurality of images from the image set generated by the image set generator based on an area of ​​application of the optical scan performed by the data set acquirer.

[0335] The image selector may be configured to divide the image set generated by the image set generator into a plurality of subsets corresponding to a plurality of sub-areas of the application area of ​​the optical scanning performed by the dataset acquirer, and further configured to select a plurality of images from the image set by selecting an image from each of the obtained subsets.

[0336] The image selector may be configured to select an image from each of the plurality of subsets based on image size.

[0337] The image selector may be configured to select from each of the plurality of subsets the largest sized image in that subset.

[0338] The image selector may be configured to, for each of the plurality of subsets, order the images included in that subset (some or all of the images included in that subset) based on image size. Further, the image selector may be configured to select images from each of the plurality of subsets based on the order assigned to the images in that subset by the ordering.

[0339] The image selector may be configured to determine whether registration is successful with respect to a first image selected from a first subset of the plurality of subsets, and may be further configured to select a second image from the first subset based on an ordering imposed on the images in the first subset if registration with respect to the first image is determined to be unsuccessful, where the second image is a different image from the first image.

[0340] The image selector may be configured to order the images included in the image set (some or all of the images included in the image set) based on image size, and may be further configured to select multiple images from the image set based on the order assigned to the images by the ordering.

[0341] The image selection unit may be configured to determine whether registration based on one image of the plurality of images selected from the image set is successful or unsuccessful. Furthermore, if it is determined that registration based on the one image is unsuccessful, the image selection unit may be configured to select a new image from the image set that is different from any of the plurality of images based on the order assigned to the images in the image set.

[0342] The ophthalmic examination apparatus 1 according to this embodiment may further include a motion information generation unit. The motion information generation unit is configured to generate motion information representing the motion of the sample when the data set acquisition unit applies an optical scan to the sample. In this embodiment, the motion information generation unit may include the data processing unit 230, the fundus camera unit 2, and / or the anterior segment camera, but these are non-limiting examples. Furthermore, the image selection unit may be configured to select multiple images from the image set based on the motion information generated by the motion information generation unit.

[0343] The image selection unit may be configured to calculate values ​​of predetermined movement parameters from the movement information generated by the movement information generation unit, evaluate the calculated values, and select multiple images from the image set based on the results of this evaluation.

[0344] The motion parameter whose value is calculated by the image selector may include at least one of the magnitude and frequency of motion. The image selector may further be configured to compare the value of the motion parameter with a predetermined threshold and select a plurality of images from a subset of the image set that excludes images based on data collected during a time period corresponding to a value equal to or greater than the threshold. The time period corresponding to a value equal to or greater than the threshold may be, by way of non-limiting example, a time period around the time of occurrence of motion whose magnitude or frequency is equal to or greater than the threshold.

[0345] The data set acquisition unit may include a deflector capable of deflecting light for optical scanning in first and second directions that are different from each other. In this embodiment, the deflector is an optical scanner 44, but this is a non-limiting example. Furthermore, to perform optical scanning for redundantly collecting data of the sample, the data set acquisition unit may be configured to repeatedly change the deflection direction in the first direction with a first period, and to repeatedly change the deflection direction in the second direction with a second period that is different from the first period.

[0346] <Second embodiment> A second embodiment will be described. In the first embodiment, a scanning imaging device configured to be able to perform optical scanning was described, but in this embodiment, an information processing device configured to receive from an external device a data set collected by optical scanning and process the data set will be described.

[0347] The information processing device of this embodiment may be configured as part of a scanning imaging device, but since such a configuration is substantially the same as the first embodiment, the following explanation will exclude such a case.

[0348] The information processing device of this embodiment may have a configuration similar to that of the scanning imaging device (ophthalmic examination device) of the first embodiment, except that it does not have a functional element that performs optical scanning (the data set acquisition unit in the first to sixth embodiments).

[0349] Any of the items described in the first embodiment can be combined with the information processing device of this embodiment, and the information processing device will achieve functions and effects according to the combined items.

[0350] The configuration of an exemplary aspect of the information processing device of this embodiment is shown in Fig. 15. The information processing device 400 of this embodiment includes a reception unit 401, an image set generation unit 402, an image selection unit 403, and an image position adjustment unit 404.

[0351] The receiving unit 401 receives a data set acquired by redundantly collecting data from a sample by optical scanning (data set receiving unit). That is, the receiving unit 401 receives a data set collected from a sample by a scanning imaging device capable of performing optical scanning such as a Lissajous scan.

[0352] The reception unit 401 has a function of acquiring data from an external device. This external device may be, for example, a computer, a storage device, a recording medium, or an information system (for example, a hospital information system, an electronic medical record system, or an image archiving system). The external device may be, for example, a device directly connected to the information processing device 400, a device connected to the information processing device 400 via a local area network (LAN), or a device connected to the information processing device 400 via a wide area network (WAN). The reception unit 401 may include, for example, a communication interface for performing data communication via a communication line and / or a drive device for reading data from a recording medium.

[0353] The image set generation unit 402 is configured to generate an image set based on the data set received by the reception unit 401. The image set generation unit 402 may have a configuration similar to that of the image set generation unit (image data construction unit 220) in the first embodiment.

[0354] The image selection unit 403 is configured to select a plurality of images from the image set generated by the image set generation unit 402. The image selection unit 403 may have a configuration similar to that of the image selection unit 225 of the first embodiment.

[0355] The image position adjustment unit 404 is configured to adjust the relative position of the image set by applying a plurality of registrations based on the plurality of images selected by the image selection unit 403 to the image set generated by the image set generation unit 402. The image position adjustment unit 404 may have a configuration similar to that of the image position adjustment unit (image data processing unit 231) of the first embodiment.

[0356] According to the information processing device 400, similarly to the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0357] 16 shows another exemplary configuration of the information processing device of this embodiment. The information processing device 500 of this embodiment includes a receiving unit 501, an image selecting unit 502, and an image position adjusting unit 503.

[0358] The receiving unit 501 receives an image set (image set receiving unit) generated based on a data set acquired by redundantly collecting data from a sample by optical scanning. That is, the receiving unit 501 receives an image set constructed from a data set collected from a sample by a scanning imaging device capable of performing optical scanning such as a Lissajous scan. The receiving unit 501 may have a configuration similar to that of the above-described receiving unit 401.

[0359] The image selection unit 502 is configured to select a plurality of images from the image set accepted by the acceptance unit 501. The image selection unit 502 may have a configuration similar to that of the image selection unit 225 in the first embodiment.

[0360] The image position adjustment unit 503 is configured to adjust the relative positions of the image set received by the reception unit 501 by applying a plurality of registrations based on the plurality of images selected by the image selection unit 502, respectively. The image position adjustment unit 503 may have a configuration similar to that of the image position adjustment unit (image data processing unit 231) of the first embodiment.

[0361] According to the information processing device 500, similarly to the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0362] <Other embodiments> The embodiments of the present disclosure are not limited to the first and second embodiments and their modifications. For example, the first and second embodiments provide embodiments relating to various methods, embodiments relating to various programs, embodiments relating to various recording media, and the like.

[0363] For example, the first embodiment described above provides an embodiment of a method for controlling a scanning imaging apparatus. This embodiment is a method for controlling a scanning imaging apparatus including a processor and a scanner that performs optical scanning. The scanner may be the data set acquisition unit of the first embodiment, and the processor may be the control unit 210, the image data construction unit 220, and the data processing unit 230 of the first embodiment.

[0364] The method for controlling a scanning imaging apparatus according to this embodiment may include the following four steps: controlling a first step to cause a scanner to redundantly collect data of a sample to acquire a data set; controlling a second step to cause a processor to generate an image set based on the data set acquired in the first step; controlling a third step to cause the processor to select multiple images from the image set generated in the second step; and controlling a fourth step to cause the processor to adjust the relative position of the image set generated in the second step by applying multiple registrations to the image set generated in the second step, each of which is based on the multiple images selected in the third step.

[0365] Any of the features described in the first or second embodiment can be combined with the control method for a scanning imaging device according to this embodiment, and the control method for a scanning imaging device will have the functions and effects according to the combined features.

[0366] According to this control method for a scanning imaging apparatus, like the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0367] The second embodiment described above provides an embodiment of a method for controlling an information processing device. This embodiment is a method for controlling an information processing device including a processor. This processor may be the control unit 210, the image data creation unit 220, and the data processing unit 230 of the first embodiment.

[0368] The control method for an information processing device according to this embodiment may include the following four steps: controlling a first step to cause a processor to accept a data set acquired by redundantly collecting data from a sample by optical scanning; controlling a second step to cause a processor to generate an image set based on the data set accepted in the first step; controlling a third step to cause the processor to select a plurality of images from the image set generated in the second step; and controlling a fourth step to cause the processor to adjust the relative position of the image set generated in the second step by applying a plurality of registrations to the image set generated in the second step, each of which is based on the plurality of images selected in the third step.

[0369] Any of the items described in the first or second embodiment can be combined with the control method of the information processing device according to this embodiment, and the control method of the information processing device will have the functions and effects according to the combined items.

[0370] According to this control method for an information processing device, similarly to the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0371] The second embodiment described above provides another embodiment of a method for controlling an information processing device. This embodiment is a method for controlling an information processing device including a processor. This processor may be the control unit 210 and the data processing unit 230 of the first embodiment.

[0372] The control method for an information processing device according to this embodiment may include the following three steps: the control of the first step causes the processor to accept an image set generated based on a data set acquired by redundantly collecting data from a sample by optical scanning; the control of the second step causes the processor to select a plurality of images from the image set accepted in the first step; and the control of the third step causes the processor to adjust the relative position of the image set accepted in the first step by applying a plurality of registrations to the image set accepted in the first step, each of which is based on the plurality of images selected in the second step.

[0373] Any of the items described in the first or second embodiment can be combined with the control method of the information processing device according to this embodiment, and the control method of the information processing device will have the functions and effects according to the combined items.

[0374] According to this control method for an information processing device, similarly to the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0375] The first and second embodiments provide embodiments of a method for processing information.

[0376] The information processing method according to this embodiment is a method for processing a data set acquired by redundantly collecting data of a sample by optical scanning, and may include the following three steps: a first step is generating an image set based on the data set; a second step is selecting a plurality of images from the image set generated in the first step; and a third step is performing relative position adjustment of the image set generated in the first step by applying a plurality of registrations based on the plurality of images selected in the second step, respectively, to the image set generated in the first step.

[0377] Any of the features described in the first or second embodiment can be combined with the information processing method according to this embodiment, and the information processing method will achieve functions and effects according to the features combined.

[0378] According to this information processing method, as in the first embodiment, it is possible to reduce errors in motion artifact correction and improve motion artifact correction.

[0379] The present disclosure provides a program for causing a computer to execute a method according to an embodiment. Some exemplary embodiments provide a program for causing a computer to execute a control method for a scanning imaging apparatus. Some exemplary embodiments provide a program for causing a computer to execute a control method for an information processing apparatus. Some exemplary embodiments provide a program for causing a computer to execute an information processing method.

[0380] The present disclosure provides a computer-readable non-transitory recording medium having a program recorded thereon according to an embodiment. Some exemplary embodiments provide a computer-readable non-transitory recording medium having a program recorded thereon that causes a computer to execute a control method for a scanning imaging apparatus. Some exemplary embodiments provide a computer-readable non-transitory recording medium having a program recorded thereon that causes a computer to execute a control method for an information processing apparatus. Some exemplary embodiments provide a computer-readable non-transitory recording medium having a program recorded thereon that causes a computer to execute an information processing method. The non-transitory recording medium of the embodiments may be in any form, examples of which include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0381] The present disclosure merely illustrates some exemplary embodiments and is not intended to limit the invention. Those who intend to implement the present invention can make any modifications (omissions, substitutions, additions, etc.) within the scope of the gist of the present invention. [Explanation of symbols]

[0382] 1. Ophthalmic examination equipment 2 Fundus camera unit 100 OCT units 210 Control Unit 220 Image Data Construction Department 225 Image Selection Section 230 Data Processing Unit 231 Image data processing unit 233 Shift amount recording section 234 Adjustment amount calculation section 235 Image data correction unit

Claims

1. A scanning imaging device that uses optical scanning to image a sample, a data set acquisition unit that acquires a data set by redundantly collecting data of the sample by the optical scanning; an image set generator that generates an image set based on the dataset; an image selector for selecting a plurality of images from the image set; an image position adjustment unit that performs relative position adjustment of the set of images by applying a plurality of registrations to the set of images, each of which is based on the plurality of images; A scanning imaging device comprising:

2. the image position adjustment unit generates a plurality of pieces of relative position information of the image set by applying the plurality of registrations to the image set, and performs the relative position adjustment based on the plurality of pieces of relative position information; 2. The scanning imaging device of claim 1.

3. the image position adjustment unit generates relative position information of the image set by applying recursive registration to the image set using one of the plurality of images as an initial reference image in each of the plurality of registrations; 3. The scanning imaging device of claim 2.

4. the image position adjustment unit determines an adjustment amount for adjusting the relative position based on the plurality of pieces of relative position information.

4. The scanning imaging device according to claim 2 or 3.

5. the image selection unit selects the plurality of images from the image set based on a time axis of the optical scanning. The scanning imaging device according to any one of claims 1 to 4.

6. the image selection unit selects the plurality of images from the image set by dividing the image set into a plurality of subsets corresponding to a plurality of sections on the time axis and selecting an image from each of the plurality of subsets.

6. The scanning imaging device of claim 5.

7. the image selector selects the plurality of images from the image set based on an application area of ​​the optical scanning; The scanning imaging device according to any one of claims 1 to 4.

8. the image selector selects the images from the image set by dividing the image set into a plurality of subsets corresponding to a plurality of sub-areas of the application area and selecting an image from each of the plurality of subsets.

8. The scanning imaging device of claim 7.

9. the image selection unit selects an image from each of the plurality of subsets based on image size; 9. The scanning imaging device according to claim 6 or 8.

10. the image selection unit selects, from each of the plurality of subsets, an image with a maximum size in the subset; 10. The scanning imaging device of claim 9.

11. the image selection unit performs ordering on the images included in each of the plurality of subsets based on image size, and selects an image from each of the plurality of subsets based on the order assigned to the images in the subset by the ordering.

10. The scanning imaging device of claim 9.

12. the image selection unit determines whether registration has been successful based on a first image selected from a first subset of the plurality of subsets, and if registration has been determined to be unsuccessful, selects a second image from the first subset based on the order assigned to the images in the first subset by the ordering.

12. The scanning imaging device of claim 11.

13. the image selection unit performs ordering on the images included in the image set based on image size, and selects the plurality of images from the image set based on the order assigned to the images by the ordering. The scanning imaging device according to any one of claims 1 to 8.

14. the image selection unit determines whether registration has been successful or not based on one of the plurality of images selected from the image set, and if it is determined to be unsuccessful, selects a new image from the image set that is different from any of the plurality of images based on the order assigned to the images in the image set by the ordering.

14. The scanning imaging device of claim 13.

15. a motion information generating unit that generates motion information representing the motion of the sample when the optical scanning is applied to the sample; the image selector selects the plurality of images from the image set based on the motion information. The scanning imaging device according to any one of claims 1 to 14.

16. the image selection unit calculates values ​​of predetermined motion parameters from the motion information, evaluates the values, and selects the plurality of images from the image set based on a result of the evaluation.

16. The scanning imaging device of claim 15.

17. the movement parameters include at least one of the magnitude and frequency of movement; the image selection unit compares the value of the motion parameter with a predetermined threshold, and selects the plurality of images from a subset obtained by excluding from the image set images based on data collected during a time period corresponding to the value equal to or greater than the threshold.

17. The scanning imaging device of claim 16.

18. The dataset acquisition unit a deflector capable of deflecting the light for the optical scanning in a first direction and a second direction different from each other, repeating a change in the deflection direction in the first direction with a first period, and repeating a change in the deflection direction in the second direction with a second period different from the first period; The scanning imaging device according to any one of claims 1 to 17.

19. a data set receiving unit that receives a data set obtained by redundantly collecting data from the sample by optical scanning; an image set generator that generates an image set based on the dataset; an image selector for selecting a plurality of images from the image set; an image position adjustment unit that performs relative position adjustment of the set of images by applying a plurality of registrations to the set of images, each of which is based on the plurality of images; An information processing device comprising:

20. an image set receiving unit that receives an image set generated based on a data set obtained by redundantly collecting data from the sample by optical scanning; an image selector for selecting a plurality of images from the image set; an image position adjustment unit that performs relative position adjustment of the set of images by applying a plurality of registrations to the set of images, each of which is based on the plurality of images; An information processing device comprising:

21. 1. A method for controlling a scanning imaging device including a processor and a scanner for optical scanning, comprising: causing the scanner to redundantly collect data of the sample to obtain a data set; causing the processor to generate an image set based on the dataset; causing the processor to select a plurality of images from the set of images; causing the processor to perform relative alignment of the set of images by applying a plurality of registrations to the set of images relative to each of the plurality of images; method.

22. A method for controlling an information processing device including a processor, comprising: causing the processor to accept a data set obtained by redundantly collecting data from a sample by optical scanning; causing the processor to generate an image set based on the dataset; causing the processor to select a plurality of images from the set of images; causing the processor to perform relative alignment of the set of images by applying a plurality of registrations to the set of images relative to each of the plurality of images; method.

23. A method for controlling an information processing device including a processor, comprising: having the processor accept an image set generated based on a data set acquired by redundantly collecting data from a sample by optical scanning; causing the processor to select a plurality of images from the set of images; causing the processor to perform relative alignment of the set of images by applying a plurality of registrations to the set of images relative to each of the plurality of images; method.

24. 1. An information processing method for processing a data set obtained by redundantly collecting data on a sample by optical scanning, comprising: generating an image set based on the dataset; selecting a plurality of images from the set of images; performing relative alignment of the set of images by applying a plurality of registrations to the set of images relative to the plurality of images; method.

25. A program that causes a computer to execute the method according to any one of claims 21 to 24.

26. A computer-readable non-transitory recording medium on which the program of claim 25 is recorded.

Images