Scanning imaging device, control method thereof, image processing device, control method thereof, scanning imaging method, image processing method, program, and recording medium

Abstract

To accelerate data processing in scan type imaging.SOLUTION: A scan type imaging device according an exemplary aspect comprises: a scan unit; an image construction unit; an image processing unit; and an image allocation unit. The scan unit is configured to collect data by applying scan according to a two-dimensional pattern including a series of cycles to a sample. The image construction unit is configured to construct a plurality of images from the collected data. The image processing unit is configured to be able to perform multithread processing. The image allocation unit is configured to allocate the plurality of images to a plurality of threads of the image processing unit. The image processing unit includes a coordinate conversion part. The coordinate conversion part is configured to execute the coordinate conversion of the plurality of images constructed by the image construction unit as at least partial parallel processing in the plurality of threads of the image processing unit.SELECTED DRAWING: Figure 4B

Description

[Technical field]

[0001] The present invention relates to a scanning imaging apparatus, a control method thereof, an image processing apparatus, a control method thereof, a scanning imaging method, an image processing method, a program, and a recording medium. [Background technology]

[0002] One imaging technique is scanning imaging, which involves sequentially irradiating a sample with a beam at multiple locations to collect data and then constructing an image of the sample from the collected data.

[0003] Optical coherence tomography (OCT) is a typical example of scanning imaging using light. OCT is a technique that can image light-scattering media with a resolution of micrometers or less, and is used in medical imaging and non-destructive testing. OCT is a technique based on low-coherence interferometry, and typically uses near-infrared light to ensure deep penetration into samples of light-scattering media.

[0004] For example, OCT devices are becoming more and more popular in ophthalmic imaging diagnosis, and not only 2D imaging but also 3D imaging, structural analysis, and functional analysis have been put to practical use, and they are now widely used as a powerful diagnostic tool. In the field of ophthalmology, scanning imaging other than OCT, such as scanning laser ophthalmoscope (SLO), is also used. Scanning imaging using light (electromagnetic waves) and ultrasound in wavelength bands other than near-infrared light is also known.

[0005] There are various scanning modes used in OCT and SLO, but the so-called "Lissajous scan" for the purpose of correcting motion artifacts has been attracting attention in recent years (see, for example, Patent Documents 1 to 4 and Non-Patent Documents 1 and 2).

[0006] In a typical Lissajous scan, the measurement light is scanned at high speed to draw multiple loops (multiple cycles that intersect with each other) of a certain size, so the difference in data acquisition time from one cycle can be practically ignored. In addition, the intersection area of ​​different cycles can be referenced to perform alignment between cycles, making it possible to correct motion artifacts caused by sample movement. Focusing on these characteristics of the Lissajous scan, the field of ophthalmology is working to address motion artifacts caused by eye movement. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2016-17915 A [Patent Document 2] JP 2018-68578 A [Patent Document 3] JP 2018-140004 A [Patent Document 4] JP 2018-140049 A [Non-patent literature]

[0008] [Non-Patent Document 1] Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Three-dimensional eye motion correction by Lissajous scan optical coherence tomography”, Biomedical Optics EXPRESS, Vol. 8, No. 3, 1 Mar 2017, PP. 1783-1802 [Non-Patent Document 2] Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Eye-motion-corrected optical coherence tomography angiography using Lissajous scanning”, Biomedical Optics EXPRESS, Vol. 9, No. 3, 1 Mar 2018, PP. 1111-1129 Summary of the Invention [Problem to be solved by the invention]

[0009] One object of the present invention is to increase the speed of data processing in scanning imaging such as a Lissajous scan. [Means for solving the problem]

[0010] A scanning imaging device according to some embodiments includes a scanning unit that applies a scan to a sample according to a two-dimensional pattern including a series of cycles to collect data, an image construction unit that constructs multiple images from the data, an image processing unit capable of multi-threaded processing, and an image allocation unit that allocates the multiple images to multiple threads of the image processing unit, and the image processing unit may include a coordinate transformation unit that performs coordinate transformation of the multiple images in the multiple threads as at least partial parallel processing.

[0011] In some aspects, for each of the plurality of images, the coordinate transformation unit may be configured to perform coordinate transformation of a plurality of pixels as a sequential process.

[0012] In some embodiments, a number of the plurality of threads may be equal to a number of the plurality of images. Furthermore, the coordinate transformation unit may be configured to perform, in each of the plurality of threads, the coordinate transformation of a plurality of pixels in one of the plurality of images as a sequential process.

[0013] In some embodiments, the scanning unit may be configured to collect the data by applying the scan to a three-dimensional region of the sample. Furthermore, the image construction unit may be configured to construct a series of two-dimensional images corresponding to the series of cycles from the data, and construct a first set of frontal images as the plurality of images by applying a depth projection to the series of two-dimensional images. Additionally, the coordinate transformation unit may be configured to transform the first set of frontal images into a second set of frontal images defined in a real space two-dimensional Cartesian coordinate system orthogonal to the depth direction.

[0014] In some aspects, the image processing unit may further include a pixel interpolation unit that performs pixel interpolation of a plurality of transformed images obtained from the plurality of images by the coordinate transformation as at least partial parallel processing.

[0015] In some aspects, for each of the plurality of images, the image processor may be configured to perform the coordinate transformation and the pixel interpolation in the same thread.

[0016] In some embodiments, the scanning portion may include a deflector capable of deflecting light in a first direction and a second direction that are distinct from one another, and the scanning portion may be configured to apply the scan to the sample by repeatedly changing the deflection direction along the first direction with a first period and repeatedly changing the deflection direction along the second direction with a second period that is different from the first period.

[0017] In some embodiments, the deflectors may be configured to be controlled based on a pre-defined scanning protocol that is based on a Lissajous function.

[0018] In some embodiments, the series of cycles may be configured to perform the scans in the two-dimensional pattern that intersect with one another.

[0019] In some embodiments, the scanning component may be configured to apply an optical coherence tomography (OCT) scan to the sample.

[0020] According to some aspects, a method for controlling a scanning imaging apparatus including a scanner configured to apply a scan to a sample to collect data and a processor may include controlling the scanner to apply a scan to the sample according to a two-dimensional pattern including a series of cycles to collect data. The method may further include controlling the processor to construct a plurality of images from the data, assign the plurality of images to a plurality of threads, and perform coordinate transformation of the plurality of images in the plurality of threads in at least partial parallel processing.

[0021] An image processing device according to some embodiments includes a memory unit that stores data collected by applying a scan to a sample according to a two-dimensional pattern including a series of cycles, an image construction unit that constructs a plurality of images from the data, an image processing unit capable of multi-threaded processing, and an image allocation unit that allocates the plurality of images to a plurality of threads of the image processing unit, and the image processing unit may include a coordinate transformation unit that performs coordinate transformation of the plurality of images in the plurality of threads as at least partial parallel processing.

[0022] According to some aspects, a method is a method for controlling an image processing device including a memory unit and a processor. The method may cause the memory unit to store data collected by scanning a sample according to a two-dimensional pattern including a series of cycles. The method may further control the processor to construct a plurality of images from the data, assign the plurality of images to a plurality of threads, and perform coordinate transformation of the plurality of images in the plurality of threads in at least partial parallel processing.

[0023] A method according to some embodiments is a scanning imaging method that collects data by applying a scan to a sample according to a two-dimensional pattern including a series of cycles, constructing a plurality of images from the data, allocating the plurality of images to a plurality of threads, and performing coordinate transformation of the plurality of images in the plurality of threads as at least partially parallel processing.

[0024] A method according to some embodiments is an image processing method that prepares collected data by applying a scan to a sample according to a two-dimensional pattern including a series of cycles, constructs a plurality of images from the data, assigns the plurality of images to a plurality of threads, and performs coordinate transformation of the plurality of images in the plurality of threads as at least partially parallel processing.

[0025] Some aspects are programs for causing a computer to execute any of the above-described methods.

[0026] Some aspects are a computer-readable non-transitory recording medium having a program recorded thereon that causes a computer to execute any of the methods described above.

[0027] Any two or more aspects can be at least partially combined. Any aspect can be at least partially combined with any feature of the present disclosure. Any partial combination of any two or more aspects can be at least partially combined with any feature of the present disclosure. Effect of the Invention

[0028] According to an exemplary embodiment, data processing in scanning imaging can be accelerated. [Brief description of the drawings]

[0029] [Figure 1] 1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Diagram 2]1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Diagram 3] 1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 4A] 1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 4B] 1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 4C] 1 is a schematic diagram illustrating an example of a configuration of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Diagram 5] 1 is a schematic diagram for explaining an example of processing executed by a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 6] 1 is a schematic diagram for explaining an example of processing executed by a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 7] 1 is a schematic diagram for explaining an example of processing executed by a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. [Figure 8] 1 is a flowchart illustrating an example of an operation of a scanning imaging apparatus (ophthalmic apparatus) according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] A scanning imaging apparatus, a control method thereof, an image processing apparatus, a control method thereof, a scanning imaging method, an image processing method, a program, and a recording medium according to some exemplary embodiments will be described with reference to the drawings.

[0031] The techniques disclosed in the documents cited in this specification and any other known techniques can be incorporated into the exemplary embodiments. In addition, unless otherwise specified, no distinction is made between "image data" and an "image" based on the image data, and no distinction is made between a "site" of the subject's eye and its "image."

[0032] In an exemplary embodiment, a scan according to a two-dimensional pattern including a series of cycles is applied to a sample. This two-dimensional pattern may typically be based on a scan protocol preset based on a Lissajous function, and may be, for example, a scan protocol used (or available) in the technology described in any of Patent Documents 1 to 4 and Non-Patent Documents 1 and 2. In the description of the exemplary embodiment, a scan mode that can be adopted will be referred to as a Lissajous scan. Hereinafter, processing of data collected by the Lissajous scan described in Non-Patent Document 1 will be described as an example.

[0033] First, a Fourier transform or the like is applied to the data collected by the Lissajous scan to construct three-dimensional image data defined in a predetermined three-dimensional Lissajous coordinate system. This three-dimensional Lissajous coordinate system is, for example, a three-dimensional Cartesian coordinate system (assumed to be an αβγ coordinate system), in which the α axis represents the positions of the multiple A-lines constituting the cycle, the β axis represents the cycle identification number (scanning order), and the γ axis represents the depth position (position in the axial direction). The three-dimensional image data expressed in the αβγ coordinate system consists of a group of cross-sectional image data (each cross section represents an αγ plane) corresponding to a group of cycles constituting the Lissajous scan, arranged in the β direction according to the scanning order.

[0034] Next, the 3D image data (volume) defined in the αβγ coordinate system is divided into multiple partial image data (sub-volumes) that do not involve relatively large motion, and an en face projection image of each sub-volume is constructed. This en face projection image is a 2D en face image defined in the αβ plane constructed by projection in the γ direction. Such an en face projection image is called a strip.

[0035] Then, each strip defined in the Lissajous coordinate system (αβ coordinate system) is transformed into an image defined in real space (xy coordinate system). In other words, a coordinate transformation from the αβ coordinate system to the xy coordinate system is applied to each strip. This process is called remapping. Furthermore, an interpolation process is applied to each remapped strip (i.e., defined in the xy coordinate system). This results in multiple strips defined in the xy coordinate system.

[0036] As a characteristic of the Lissajous scan described in Non-Patent Document 1, the two strips overlap (intersect) at four points. Therefore, in the data processing described in Non-Patent Document 1, the largest strip is adopted as an initial reference strip, and the strips are sequentially registered and merged in order of size with respect to this initial reference strip. In the registration, a correlation calculation (cross-correlation function, correlation coefficient) is used to determine the relative position between the reference strip and other strips. Here, since the strips have any shape, a correlation calculation is performed on the overlapping area between the strips using a mask for treating each strip as an image of a specific shape (for example, a square shape) (see Appendix A of Non-Patent Document 1). As a result, a merged image (composite image) of multiple strips is obtained. This merged image is the target image, that is, an image in which motion artifacts have been corrected.

[0037] In the exemplary embodiment, the remapping and interpolation processes are devised to speed up (reduce) data processing.

[0038] In remapping, multiple pixels of a strip defined in the Lissajous coordinate system may be converted to the same pixel in the xy coordinate system. In other words, remapping has output dependency. Therefore, data parallel processing cannot be applied to the remapping of a single strip, and serial processing must be applied.

[0039] In this way, although the remapping of each strip cannot be performed in parallel, it is possible to parallelize the entire remapping. That is, it is possible to perform the remapping of a plurality of strips in parallel. For example, task parallel processing (multi-thread processing) can be applied with the number of threads equal to or less than the number of strips. Here, by performing task parallel processing with the number of threads equal to the number of strips, the remapping can be performed at the highest speed. This is one of the findings that the present inventor has found from the study and consideration of data processing in scanning imaging.

[0040] Another finding made by the inventor is related to the interpolation process. The interpolation process is a data process in which a pixel to which no value is assigned in the remapping process is assigned a value derived from the values ​​of the surrounding pixels. Therefore, there is a possibility that output dependency may occur. Therefore, as with the remapping process, the interpolation process for each strip cannot be performed in parallel, but the interpolation process for multiple strips can be performed in parallel.

[0041] Several exemplary embodiments conceived based on these novel findings will be described below.

[0042] The scanning imaging device according to the exemplary embodiment described below is an ophthalmologic device capable of measuring the fundus of a living eye using Fourier domain OCT (e.g., swept source OCT). The type of OCT that can be adopted in the embodiment is not limited to swept source OCT, and may be, for example, spectral domain OCT or time domain OCT.

[0043] Scanning imaging devices according to some exemplary embodiments may be capable of utilizing scanning modalities other than OCT, for example, some exemplary embodiments may employ any optical scanning modality, such as SLO.

[0044] In addition, the scanning modality applicable to the scanning imaging apparatus according to some exemplary embodiments is not limited to the optical scanning modality. For example, some exemplary embodiments may employ a scanning modality using electromagnetic waves other than light, a scanning modality using ultrasound, or the like.

[0045] In addition to processing data collected by OCT scans and / or processing data collected by SLO, the ophthalmic device according to some exemplary embodiments may be capable of processing data acquired by other modalities, such as, for example, a fundus camera, a slit lamp microscope, and an ophthalmic surgical microscope. The ophthalmic device according to some exemplary embodiments may have the functionality of such modalities.

[0046] The object (sample) to which OCT is applied is not limited to the fundus, but may be any part of the eye, such as the anterior segment or vitreous body, or a part or tissue of a living body other than the eye, or an object other than a living body.

[0047] The ophthalmic device according to the exemplary embodiment described below provides some aspects of a scanning imaging device, some aspects of a method for controlling a scanning imaging device, some aspects of an image processing device, some aspects of a method for controlling an image processing device, some aspects of an image processing method, and some aspects of a scanning imaging method.

[0048] Configuration of Scanning Imaging Device The ophthalmic apparatus 1 shown in FIG. 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic and control unit 200. The fundus camera unit 2 includes a group of elements (optical elements, mechanisms, etc.) for photographing the subject's eye E from the front. The OCT unit 100 includes a part of the group of elements (optical elements, mechanisms, etc.) for applying an OCT scan to the subject's eye E. The other part of the group of elements for the OCT scan is provided in the fundus camera unit 2. The arithmetic and control unit 200 includes one or more processors that execute various calculations and controls. In addition to these, the ophthalmic apparatus 1 may include an element for supporting the subject's face and an element for switching the part to which the OCT scan is applied. Examples of the former element include a chin rest and a forehead rest. Examples of the latter element include a lens unit used to switch the part to which the OCT scan is applied from the fundus to the anterior segment.

[0049] The functions of the elements disclosed herein may be implemented using circuitry or processing circuitry. Circuitry or processing circuitry includes any of a general purpose processor, a special purpose processor, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), conventional circuitry, and any combination thereof, configured and / or configured to perform the disclosed functions. A processor is considered to be processing circuitry or circuitry, including transistors and / or other circuitry. In this disclosure, circuitry, unit, means, or similar terms are hardware that performs the disclosed functions or is hardware that is programmed to perform the disclosed functions. The hardware may be hardware disclosed herein or may be known hardware that is programmed and / or configured to perform the described functions. In the case where the hardware is a processor, which may be considered to be a type of circuitry, the circuitry, unit, means, or similar terms are a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0050] Fundus Camera Unit 2 The fundus camera unit 2 is provided with an optical system for photographing the fundus Ef of the subject's eye E. The acquired image of the fundus Ef (called a fundus image, fundus photograph, etc.) is a front image such as an observed image or a photographed image. The observed image is obtained by, for example, video shooting using near-infrared light, and is used for alignment, focusing, tracking, etc. The photographed image is a still image using, for example, a flash light in the visible or infrared range.

[0051] The fundus camera unit 2 includes an illumination optical system 10 and an imaging optical system 30. The illumination optical system 10 irradiates illumination light onto the subject's eye E. The imaging optical system 30 detects return light of the illumination light from the subject's eye E. The measurement light from the OCT unit 100 is guided to the subject's eye E through an optical path in the fundus camera unit 2, and the return light is guided to the OCT unit 100 through the same optical path.

[0052] Light (observation illumination light) output from the observation light source 11 of the illumination optical system 10 is reflected by the concave mirror 12, passes through the condenser lens 13, and passes through the visible cut filter 14 to become near-infrared light. Furthermore, the observation illumination light is once focused near the photographing light source 15, reflected by the mirror 16, and passes through the relay lens system 17, the relay lens 18, the diaphragm 19, and the relay lens system 20. The observation illumination light is then reflected at the periphery (area around the hole) of the aperture mirror 21, passes through the dichroic mirror 46, and is refracted by the objective lens 22 to illuminate the subject's eye E (fundus Ef). Return light of the observation illumination light from the subject's eye E is refracted by the objective lens 22, passes through the dichroic mirror 46, passes through the hole formed in the central area of ​​the aperture mirror 21, passes through the dichroic mirror 55, passes through the photographing focusing lens 31, and is reflected by the mirror 32. Furthermore, this return light passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged on the light receiving surface of the image sensor 35 by the imaging lens 34. The image sensor 35 detects the return light at a predetermined frame rate. The focus (focal position) of the photographing optical system 30 is adjusted to match the fundus Ef or the anterior segment of the eye.

[0053] The light (imaging illumination light) output from the imaging light source 15 is irradiated onto the fundus Ef through the same path as the observation illumination light. The return light of the imaging illumination light from the subject's eye E is guided to the dichroic mirror 33 through the same path as the return light of the observation illumination light, passes through the dichroic mirror 33, is reflected by a mirror 36, and is imaged by an imaging lens 37 on the light receiving surface of an image sensor 38.

[0054] A liquid crystal display (LCD) 39 displays a fixation target (fixation target image). A part of the light beam output from the LCD 39 is reflected by the half mirror 33A, reflected by the mirror 32, passes through the photographing focusing lens 31 and the dichroic mirror 55, and passes through the hole of the aperture mirror 21. The light beam that passes through the hole of the aperture mirror 21 passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the fundus Ef. By changing the display position of the fixation target image on the LCD 39, the direction (fixation direction, fixation position) in which the gaze of the subject's eye E is guided is changed.

[0055] Instead of a display device such as an LCD, for example, a light-emitting element array, or a combination of a light-emitting element and a mechanism for moving the light-emitting element may be used.

[0056] The alignment optical system 50 generates an alignment index used for positioning (alignment) of the optical system with respect to the subject's eye E. Alignment light output from a light-emitting diode (LED) 51 passes through an aperture 52, an aperture 53, and a relay lens 54, is reflected by a dichroic mirror 55, passes through a hole in the aperture mirror 21, transmits through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. Return light (corneal reflected light, etc.) of the alignment light from the subject's eye E is guided to the image sensor 35 via the same path as the return light of the observation illumination light. Manual alignment or auto alignment can be performed based on the received light image (alignment index image).

[0057] As in the conventional example, the alignment index image in this example is composed of two bright spot images whose positions change depending on the alignment state. When the relative position between the subject's eye E and the optical system changes in the xy direction, the two bright spot images are displaced in the xy direction as a unit. When the relative position between the subject's eye E and the optical system changes in the z direction, the relative position (distance) between the two bright spot images changes. When the distance between the subject's eye E and the optical system in the z direction matches a predetermined working distance, the two bright spot images overlap. When the position of the subject's eye E and the optical system match in the xy direction, the two bright spot images are presented in or near a predetermined alignment target. When the distance between the subject's eye E and the optical system in the z direction matches the working distance and the position of the subject's eye E and the optical system match in the xy direction, the two bright spot images are presented in the alignment target while overlapping.

[0058] In the automatic alignment, the data processing unit 230 detects the positions of the two bright spot images, and the main control unit 211 controls the moving mechanism 150 (described later) based on the positional relationship between the two bright spot images and the alignment target. In the manual alignment, the main control unit 211 displays the two bright spot images on the display unit 241 together with the observation image of the subject's eye E, and the user operates the moving mechanism 150 using the operation unit 242 while referring to the two displayed bright spot images.

[0059] The focusing optical system 60 generates a split index used for focus adjustment of the subject's eye E. In conjunction with the movement of the photographing focusing lens 31 along the optical path (photographing optical path) of the photographing optical system 30, the focusing optical system 60 is moved along the optical path (illumination optical path) of the illumination optical system 10. The reflecting rod 67 is inserted and removed from the illumination optical path. When performing focus adjustment, the reflecting surface of the reflecting rod 67 is tilted and disposed on the illumination optical path. The focusing light output from the LED 61 passes through the relay lens 62, is split into two light beams by the split index plate 63, passes through the two-hole diaphragm 64, is reflected by the mirror 65, and is once imaged and reflected on the reflecting surface of the reflecting rod 67 by the condenser lens 66. Furthermore, the focusing light passes through the relay lens 20, is reflected by the aperture mirror 21, passes through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. The return light (fundus reflection light, etc.) of the focusing light from the subject's eye E passes through the same path as the return light of the alignment light and is guided to the image sensor 35. Based on the received light image (split target image), manual focusing or autofocusing can be performed.

[0060] Diopter correction lenses 70 and 71 can be selectively inserted into the photographing optical path between the aperture mirror 21 and the dichroic mirror 55. The diopter correction lens 70 is a plus lens (convex lens) for correcting strong hyperopia. The diopter correction lens 71 is a minus lens (concave lens) for correcting strong myopia.

[0061] The dichroic mirror 46 combines the optical path for fundus imaging and the optical path for OCT (measurement arm). The dichroic mirror 46 reflects light in the wavelength band used for OCT and transmits light for fundus imaging. The measurement arm is provided with a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45 in this order from the OCT unit 100 side.

[0062] The retroreflector 41 is movable in the direction of the arrow shown in Fig. 1, thereby changing the length of the measurement arm. The change in the measurement arm length is used, for example, to correct the optical path length according to the axial length, adjust the interference state, and so on.

[0063] The dispersion compensation member 42, together with a dispersion compensation member 113 (described later) disposed in the reference arm, acts to match the dispersion characteristics of the measurement light LS with the dispersion characteristics of the reference light LR.

[0064] The OCT focusing lens 43 is moved along the measurement arm to adjust the focus of the measurement arm. The movement of the imaging focusing lens 31, the movement of the focus optical system 60, and the movement of the OCT focusing lens 43 can be controlled in a coordinated manner.

[0065] The optical scanner 44 is disposed at a position that is substantially optically conjugate with the pupil of the subject's eye E. The optical scanner 44 deflects the measurement light LS guided by the measurement arm. The optical scanner 44 is, for example, a galvanometer scanner capable of two-dimensional scanning, including a galvanometer mirror for scanning in the x direction and a galvanometer mirror for scanning in the y direction.

[0066] <OCT Unit 100> 2, the OCT unit 100 is provided with an optical system for applying swept-source OCT. This optical system includes an interference optical system. This interference optical system splits light from a wavelength-variable light source (wavelength-swept light source) into measurement light and reference light, superimposes the return light of the measurement light from the subject's eye E and the reference light that has passed through a reference light path to generate interference light, and detects this interference light. Data (detection signal) obtained by the interference optical system is a signal that represents the spectrum of the interference light, and is sent to the arithmetic and control unit 200.

[0067] The light source unit 101 includes, for example, a near-infrared tunable laser that changes the emission wavelength at high speed at least in the near-infrared wavelength band. The light L0 output from the light source unit 101 is guided to a polarization controller 103 by an optical fiber 102, where its polarization state is adjusted. Furthermore, the light L0 is guided to a fiber coupler 105 by an optical fiber 104, where it is split into a measurement light LS and a reference light LR. The optical path of the measurement light LS is called a measurement arm, and the optical path of the reference light LR is called a reference arm.

[0068] The reference light LR is guided by an optical fiber 110 to a collimator 111, where it is converted into a parallel beam, and is guided to a retroreflector 114 via an optical path length correction member 112 and a dispersion compensation member 113. The optical path length correction member 112 acts to match the optical path length of the reference light LR with that of the measurement light LS. The dispersion compensation member 113 acts, together with a dispersion compensation member 42 arranged in the measurement arm, to match the dispersion characteristics between the reference light LR and the measurement light LS. The retroreflector 114 is movable along the optical path of the reference light LR incident thereon, thereby changing the length of the reference arm. The change in the reference arm length is used, for example, for correcting the optical path length according to the axial length of the eye, adjusting the interference state, and the like.

[0069] The reference light LR that has passed through the retroreflector 114 passes through the dispersion compensation member 113 and the optical path length correction member 112, is converted from a parallel beam to a focused beam by the collimator 116, and enters an optical fiber 117. The reference light LR that has entered the optical fiber 117 is guided to a polarization controller 118 where its polarization state is adjusted, is guided through an optical fiber 119 to an attenuator 120 where its light amount is adjusted, and is guided through an optical fiber 121 to a fiber coupler 122.

[0070] On the other hand, the measurement light LS generated by the fiber coupler 105 is guided by the optical fiber 127 and converted into a parallel beam by the collimator lens unit 40, passes through the retroreflector 41, the dispersion compensation member 42, the OCT focusing lens 43, the optical scanner 44, and the relay lens 45, is reflected by the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. The measurement light LS is scattered and reflected at various depth positions in the subject's eye E. The return light of the measurement light LS from the subject's eye E travels in the opposite direction along the same path as the outward path and is guided to the fiber coupler 105, and reaches the fiber coupler 122 via the optical fiber 128.

[0071] The fiber coupler 122 generates interference light by superimposing the measurement light LS incident via the optical fiber 128 and the reference light LR incident via the optical fiber 121. The fiber coupler 122 splits the generated interference light at a predetermined splitting ratio (for example, 1:1) to generate a pair of interference lights LC. The pair of interference lights LC are guided to a detector 125 via optical fibers 123 and 124, respectively.

[0072] The detector 125 includes, for example, a balanced photodiode. The balanced photodiode has a pair of photodetectors that detect a pair of interference lights LC, respectively, and outputs the difference between the pair of detection results obtained by these. The detector 125 sends this output (detection signal) to a data acquisition system (DAS) 130.

[0073] A clock KC is supplied from the light source unit 101 to the data collection system 130. The clock KC is generated in the light source unit 101 in synchronization with the output timing of each wavelength swept within a predetermined wavelength range by the wavelength-tunable light source. For example, the light source unit 101 branches the light L0 of each output wavelength to generate two branched lights, optically delays one of the branched lights, combines the branched lights, detects the obtained combined light, and generates the clock KC based on the detection result. The data collection system 130 samples the detection signal input from the detector 125 based on the clock KC. The data collection system 130 sends the result of this sampling to the arithmetic and control unit 200.

[0074] In this example, both an element for changing the measurement arm length (e.g., retroreflector 41) and an element for changing the reference arm length (e.g., retroreflector 114 or reference mirror) are provided, but only one of the elements may be provided. In addition, the element for changing the difference (optical path length difference) between the measurement arm length and the reference arm length is not limited to these, and may be any element (optical member, mechanism, etc.).

[0075] <Control system / processing system> 3, 4A, 4B, and 4C show configuration examples of the control system and processing system of the ophthalmic apparatus 1. The control unit 210, the image construction unit 220, and the data processing unit 230 are provided in, for example, an arithmetic control unit 200. The ophthalmic apparatus 1 may include a communication device for performing data communication with an external device. The ophthalmic apparatus 1 may include a drive device (reader / writer) for reading data from a recording medium and writing data to the recording medium.

[0076] <Control unit 210> The control unit 210 executes various controls. The control unit 210 includes a main control unit 211 and a storage unit 212. In this embodiment, as shown in FIG. 4A, the main control unit 211 includes a scanning control unit 2111, and the storage unit 212 stores a scanning protocol 2121.

[0077] <Main control unit 211> The main controller 211 includes a processor and controls each element (including the elements shown in Figs. 1 to 4C) of the ophthalmologic apparatus 1. The main controller 211 is realized by cooperation between hardware including the processor and control software. The scan controller 2111 controls an OCT scan for a scan area of ​​a predetermined shape and a predetermined size.

[0078] The imaging focusing driver 31A moves the imaging focusing lens 31 arranged in the imaging optical path and the focus optical system 60 arranged in the illumination optical path under the control of the main controller 211. The retroreflector (RR) driver 41A moves the retroreflector 41 arranged in the measurement arm under the control of the main controller 211. The OCT focusing driver 43A moves the OCT focusing lens 43 arranged in the measurement arm under the control of the main controller 211. The retroreflector (RR) driver 114A arranged in the measurement arm moves the retroreflector 114 arranged in the reference arm under the control of the main controller 211. Each of the above-mentioned drivers includes an actuator such as a pulse motor that operates under the control of the main controller 211. The optical scanner 44 operates under the control of the main controller 211 (scanning controller 2111).

[0079] The movement mechanism 150 typically moves the fundus camera unit 2 three-dimensionally, and includes, for example, an x-stage movable in ±x directions (left and right directions), an x ​​movement mechanism for moving the x-stage, a y-stage movable in ±y directions (up and down directions), a y movement mechanism for moving the y-stage, a z-stage movable in ±z directions (depth direction), and a z movement mechanism for moving the z-stage. Each movement mechanism includes an actuator such as a pulse motor that operates under the control of the main controller 211.

[0080] <Storage section 212> The storage unit 212 stores various types of data. The data stored in the storage unit 212 include an OCT image, a fundus image, subject's eye information, and control information. The subject's eye information includes subject information such as a patient ID and name, left / right eye identification information, and electronic medical record information. The control information is information related to a specific control. The control information in this embodiment includes a scan protocol 2121.

[0081] The scanning protocol 2121 is an agreement on the content of control for an OCT scan for a scanning area of ​​a predetermined shape and a predetermined size, and includes a set of various control parameters (scanning control parameters). The scanning protocol 2121 includes a protocol for each scanning mode. The scanning protocol 2121 of this embodiment includes at least a protocol for a Lissajous scan, and may further include protocols for, for example, a B scan (line scan), a cross scan, a radial scan, and a raster scan.

[0082] The scan control parameters of this embodiment include at least a parameter indicating the content of control over the optical scanner 44. These parameters include, for example, a parameter indicating a scan pattern, a parameter indicating a scan speed, and a parameter indicating a scan interval. The scan pattern indicates the shape of the scan path, and examples of such a pattern include a Lissajous pattern, a line pattern, a cross pattern, a radial pattern, and a raster pattern. The scan speed is defined, for example, as the repetition rate of A-scans. The scan interval is defined, for example, as the interval between adjacent A-scans, that is, the arrangement interval of scan points.

[0083] In addition, similar to the conventional techniques disclosed in Patent Documents 1 to 4 and Non-Patent Documents 1 and 2, the "Lissajous scan" of this embodiment may be not only a "narrowly defined" Lissajous scan whose path is a pattern drawn by the locus of points obtained by forming an ordered pair of two mutually perpendicular simple harmonic motions (Lissajous pattern, Lissajous figure, Lissajous curve, Lissajous function, Bowditch curve), but also a "broadly defined" Lissajous scan that follows a predetermined two-dimensional pattern including a series of cycles.

[0084] In this embodiment, the optical scanner 44 includes a first galvanometer mirror that deflects the measurement light LS in the x direction and a second galvanometer mirror that deflects the measurement light LS in the y direction. The Lissajous scan is realized by controlling the first galvanometer mirror so as to repeatedly change the deflection direction along the x direction in a first period, while controlling the second galvanometer mirror so as to repeatedly change the deflection direction along the y direction in a second period. Here, the first period and the second period are different from each other.

[0085] For example, the Lissajous scan of this embodiment may be not only a scan of a narrow Lissajous pattern obtained by combining two sine waves, but also a scan of a pattern to which a specific term (e.g., an odd-order polynomial) has been added, or a scan of a pattern based on a triangular wave.

[0086] A "cycle" generally means an object consisting of multiple sampling points having a certain length, which may be, for example, a closed or nearly closed curve (i.e., the start and end points of the cycle may coincide or nearly coincide).

[0087] Typically, the scanning protocol 2121 is set based on the Lissajous function. As shown in equation (9) of Non-Patent Document 1, the Lissajous function is expressed, for example, by the following parametric equation system: x(t i )=A cos(2π (f A / n)·t i ), y(t i )=A cos(2π (f A (n-2) / n 2 )·t i ).

[0088] Here, x is the horizontal axis of the two-dimensional coordinate system in which the Lissajous curve is defined, y is the vertical axis, and t i is the acquisition time of the i-th A-line in the Lissajous scan, A is the scan range (amplitude), f Aindicates the A-line collection rate (scan speed, A-scan repetition rate), and n indicates the number of A-lines in each cycle in the x (horizontal axis) direction.

[0089] An example of the distribution of scan lines applied to the fundus Ef in such a Lissajous scan is shown in FIG.

[0090] Since any pair of cycles included in a (narrow or broad) Lissajous scan crosses each other at one or more points (particularly two or more points, typically four points), it is possible to perform registration between pairs of data collected from any pair of cycles in the Lissajous scan. This makes it possible to use the image construction method and motion artifact correction method disclosed in Non-Patent Document 1 (or 2). Hereinafter, unless otherwise specified, the method described in Non-Patent Document 1 is applied.

[0091] The control information stored in the storage unit 212 is not limited to the above example. For example, the control information may include information for performing focus control (focus control parameters).

[0092] The focus control parameters are parameters indicating the content of control over the OCT focusing drive unit 43A. Examples of focus control parameters include a parameter indicating the focal position of the measurement arm, a parameter indicating the moving speed of the focal position, and a parameter indicating the moving acceleration of the focal position. The parameter indicating the focal position is, for example, a parameter indicating the position of the OCT focusing lens 43. The parameter indicating the moving speed of the focal position is, for example, a parameter indicating the moving speed of the OCT focusing lens 43. The parameter indicating the moving acceleration of the focal position is, for example, a parameter indicating the moving acceleration of the OCT focusing lens 43. The moving speed may or may not be constant. The same applies to the moving acceleration.

[0093] Such focus control parameters allow focus adjustment according to the shape of the fundus Ef (typically a concave shape with a deep center and a shallow periphery) and aberration distribution. Focus control is performed in conjunction with scan control (repetitive control of Lissajous scan), for example. This allows motion artifacts to be corrected and a high-quality image that is in focus over the entire scan range to be obtained.

[0094] <Scanning control unit 2111> The scanning control unit 2111 controls at least the optical scanner 44 based on the scanning protocol 2121. The scanning control unit 2111 may further control the light source unit 101 in cooperation with the control of the optical scanner 44 based on the scanning protocol 2121. The scanning control unit 2111 is realized by cooperation between hardware including a processor and scanning control software including the scanning protocol 2121.

[0095] <Image construction unit 220> The image constructing unit 220 includes a processor and constructs OCT image data of the fundus Ef based on a signal (sampling data) input from the data acquisition system 130. This OCT image data construction includes noise removal (noise reduction), filtering, fast Fourier transform (FFT), and the like, similar to conventional Fourier domain OCT (swept source OCT). When another type of OCT method is adopted, the image constructing unit 220 constructs OCT image data by performing known processing according to the type. This results in multiple A-scan image data. Coordinates according to the scanning protocol are assigned to these A-scan image data.

[0096] As described above, in this embodiment, a Lissajous scan is applied to the fundus Ef. The image construction unit 220 constructs three-dimensional image data of the fundus Ef by applying, for example, an image construction method and a motion artifact correction method disclosed in Non-Patent Documents 1 or 2 to data acquired through collection by the Lissajous scan and sampling by the data collection system 130. In some exemplary aspects, the image construction unit 220, together with the data processing unit 230, executes construction of three-dimensional image data of the fundus Ef.

[0097] The image constructor 220 and / or the data processor 230 may apply rendering to the 3D image data to form an image for display. Examples of applicable rendering techniques include volume rendering, surface rendering, maximum intensity projection (MIP), minimum intensity projection (MIP), multiplanar reconstruction (MPR), etc.

[0098] The image constructing unit 220 and / or the data processing unit 230 can construct an enface OCT image based on the three-dimensional image data. For example, the image constructing unit 220 and / or the data processing unit 230 can construct projection data by projecting the three-dimensional image data in the z direction (A-line direction, depth direction). The image constructing unit 220 and / or the data processing unit 230 can also construct a shadowgram by projecting partial three-dimensional image data, which is a part of the three-dimensional image data, in the z direction. This partial three-dimensional image data is set, for example, by using segmentation. Segmentation is a process of identifying a partial region in an image. In this example, segmentation can be performed to identify an image region corresponding to a specific tissue of the fundus Ef.

[0099] The ophthalmologic apparatus 1 may be capable of performing OCT angiography. OCT angiography is an imaging technique that constructs an image in which blood vessels are emphasized (see, for example, Non-Patent Document 2, JP2015-515894A, etc.). In general, fundus tissue (structure) does not change over time, but blood flow parts inside blood vessels change over time. In OCT angiography, an image is generated by emphasizing parts (blood flow signals) in which such temporal changes exist. Note that OCT angiography is also called OCT motion contrast imaging. Also, images acquired by OCT angiography are called angiography images, angiograms, motion contrast images, etc.

[0100] When OCT angiography is performed, the ophthalmic apparatus 1 repeatedly scans the same area of ​​the fundus Ef a predetermined number of times. For example, the ophthalmic apparatus 1 repeatedly performs the above-mentioned scanning control (repeated control of Lissajous scan) a predetermined number of times. As a result, a plurality of three-dimensional data (three-dimensional data set) corresponding to the application area of ​​the Lissajous scan is collected by the data collection system 130. The image construction unit 220 and / or the data processing unit 230 can construct a motion contrast image from this three-dimensional data set. This motion contrast image is an angiographic image in which the temporal change of the interference signal caused by the blood flow of the fundus Ef is emphasized. This angiographic image is three-dimensional angiographic image data representing the three-dimensional distribution of blood vessels of the fundus Ef.

[0101] The image constructing unit 220 and / or the data processing unit 230 can construct any two-dimensional angiographic image data and / or any pseudo three-dimensional angiographic image data from this three-dimensional angiographic image data. For example, the image constructing unit 220 can construct two-dimensional angiographic image data representing any cross section of the fundus Ef by applying multiplanar reconstruction to the three-dimensional angiographic image data. The image constructing unit 220 can also construct frontal angiographic image data of the fundus Ef by applying projection imaging or shadowgramming to the three-dimensional angiographic image data.

[0102] In this embodiment, the image construction unit 220 is configured to construct multiple strips defined in a predetermined Lissajous coordinate system (e.g., the aforementioned αβ coordinate system) from data collected by the data collection system 130, generate multiple strips defined in a real space coordinate system (e.g., the aforementioned xy coordinate system) by applying coordinate transformation to each constructed strip, and apply interpolation processing to each generated strip.

[0103] An example of the configuration of the image constructing unit 220 for performing such processing is shown in Fig. 4B. The image constructing unit 220 in this example includes a fast Fourier transform (FFT) unit 221, a strip constructing unit 222, an image allocation unit 223, and an image processing unit 224. Unless otherwise specified, this example follows the technology described in Non-Patent Document 1.

[0104] <FFT section 221> The FFT unit 221 constructs three-dimensional image data (volume) by applying a fast Fourier transform (FFT) or the like to data collected by a Lissajous scan of a three-dimensional region of the subject's eye E. Note that a predetermined process (pre-processing and / or post-processing) can be performed before and / or after the fast Fourier transform.

[0105] Strip construction unit 222 The strip construction unit 222 first divides the volume constructed by the FFT unit 221 into a plurality of sub-volumes in which relatively large motion does not occur. Furthermore, the strip construction unit 222 constructs a front projection image of each sub-volume. This front projection image is a strip defined in the Lissajous coordinate system.

[0106] FIG. 6 shows an example of a strip defined in a Lissajous coordinate system (the aforementioned αβ coordinate system). Reference numeral 300 denotes a front projection image of the entire volume. The entire volume is defined in an αβγ coordinate system, and the front projection image 300 obtained by projecting this in the γ direction is also defined in the αβ coordinate system. Here, the α axis represents the positions of the multiple A-lines that make up the cycle, and the β axis perpendicular to this represents the identification number (scanning order) of the cycle. Furthermore, the γ axis perpendicular to the αβ plane represents the depth direction. Typically, both the γ direction and the z direction coincide with the direction along the A-lines of the OCT scan (the traveling direction of the measurement light LS).

[0107] 6 shows N strips 300(0) to 300(N-1), where N is an integer equal to or greater than 2 and typically ranges from tens to hundreds. Each strip 300(n) is a partial image of a front projection image 300 of the entire volume.

[0108] <Image allocation unit 223 and image processing unit 224> The image allocation unit 223 is configured to allocate the plurality of strips 300(0) to 300(N−1) constructed by the strip construction unit 222 to the plurality of threads set in the image processing unit 224.

[0109] Here, the image processing unit 224 is capable of multi-thread processing and includes, for example, a GPU, that is, the exemplary image processing unit 224 includes a processor capable of operating as a general-purpose computing on GPU (GPGPU).

[0110] For example, the image allocation unit 223 may be configured to set multiple threads in the image processing unit 224, or to select multiple threads from a group of threads preset in the image processing unit 224. The number of threads may be any number greater than or equal to 2 and less than or equal to the number of strips (N).

[0111] Typically, the number of threads is equal to the number (N) of strips. This allows all N strips to be processed in parallel. In the image processing unit 224 shown in FIG. 4B, N threads 224(0) to 224(N-1) are set, the number of threads being equal to the number (N) of strips. The image allocation unit 223 in this example allocates the strips 300(0) to 300(N-1) constructed by the strip construction unit 222 to the threads 224(0) to 224(N-1) of the image processing unit 224, respectively. Here, the strip 300(n) is allocated to the thread 224(n) (n=0, 1, . . . , N-1).

[0112] As shown in FIG. 4C, for each n=0, 1, . . . , N-1, thread 224(n) includes a coordinate conversion unit 225(n) and a pixel interpolation unit 226(n).

[0113] <Coordinate conversion unit 225(n)> The coordinate conversion unit 225(n) is configured to perform coordinate conversion of the strip 300(n) assigned to the thread 224(n). In this example, the N coordinate conversion units 225(0) to 225(N-1) in the N threads 224(0) to 224(N-1) can perform all of the coordinate conversion of the N strips 300(0) to 300(N-1) as parallel processing.

[0114] As described above, in general, the number of threads is equal to or greater than 2 and equal to or less than N. When the number of threads is N as in this example, all of the N coordinate transformation units 225(0) to 225(N-1) can execute processes in parallel.

[0115] On the other hand, when the number of threads is 2 or more and N-1 or less, the coordinate transformation of N strips is typically executed as "partial parallel processing". That is, some of the N strips are executed as parallel processing, and the other part is executed as sequential processing. For example, when N is an even number, the number of threads is N / 2, and two strips are assigned to each thread, the coordinate transformation of a first group of strips (N / 2 strips) among the N strips can be executed as parallel processing, and then the coordinate transformation of a second group of strips (the remaining N / 2 strips) can be executed as parallel processing. Here, the coordinate transformation of the first group of strips and the coordinate transformation of the second group of strips can be said to be sequential processing. That is, in each thread, the coordinate transformation of two strips is executed as sequential processing.

[0116] The coordinate transformation is a coordinate transformation from a Lissajous coordinate system (for example, the aforementioned αβ coordinate system) to a real space coordinate system (for example, the aforementioned xy coordinate system), and corresponds to the remapping described in Non-Patent Document 1.

[0117] The process of creating a group of strips defined in a real space coordinate system from the volume constructed by the FFT unit 221 is not limited to the process in this example. For example, a front projection image of the entire volume can be constructed, and this front projection image can be divided to construct multiple strips (defined in a Lissajous coordinate system), and coordinate transformation can be applied to these. Alternatively, a front projection image of the entire volume can be constructed, and this front projection image (defined in a Lissajous coordinate system) can be applied to coordinate transformation, and the transformed front projection image (defined in a real space coordinate system) can be divided to construct multiple strips.

[0118] In the present disclosure, "parallel processing" means that multiple processes are executed in parallel in terms of time. Here, a first process and a second process being parallel means that at least a part of the execution period of the first process and at least a part of the execution period of the second process overlap in terms of time. Therefore, "parallel processing" in the present disclosure can be said to be executing multiple processes at least partially simultaneously.

[0119] <Pixel Interpolation Unit 226(n)> The pixel interpolation unit 226(n) is configured to interpolate pixels of an image (transformed image) obtained from the strip 300(n) by the coordinate conversion unit 225(n). Here, the transformed image is a strip (also referred to as a strip) defined in a real space coordinate system (for example, the above-mentioned xy coordinate system).

[0120] In this example, the N pixel interpolation units 226(0) to 226(N-1) in the N threads 224(0) to 224(N-1) can execute all of the pixel interpolation of the N strips obtained from the N strips 300(0) to 300(N-1) as parallel processing.

[0121] When the number of threads is N as in this example, all of the N pixel interpolation units 226(0) to 226(N-1) can execute processing in parallel. On the other hand, when the number of threads is 2 or more and N-1 or less, pixel interpolation of N strips is executed as partial parallel processing.

[0122] The pixel interpolation may include, for example, any known pixel interpolation operation. For example, the pixel interpolation unit 226 first identifies pixels of the strip obtained from the strip 300(n) by the coordinate conversion unit 225(n) that are not assigned values ​​from the pixels of the strip 300(n) (in other words, pixels that are not included in the range of values ​​of the coordinate conversion). The identified pixels are then subject to the interpolation.

[0123] Next, the pixel interpolation unit 226 identifies pixels within a predetermined vicinity of the pixel to be interpolated that have been assigned values ​​from the pixels of strip 300(n) (in other words, pixels included in the range of coordinate transformation), calculates the value of the pixel to be interpolated from the values ​​of these pixels, and assigns the value to the pixel to be interpolated.

[0124] Here, the predetermined neighborhood area is arbitrary, and the calculation for calculating the value of the interpolation target pixel from the values ​​of the pixels inside the predetermined neighborhood area is also arbitrary. For example, the average of the values ​​of the pixels included in the value range of the coordinate conversion among the pixels adjacent to the interpolation target pixel (typically, the eight pixels located above, below, right, left, upper right, lower right, upper left, and lower left of the interpolation target pixel) can be calculated, and this average value can be used as the value of the interpolation target pixel.

[0125] In this example, each thread 224(n) includes both a coordinate transformation unit 225(n) and a pixel interpolation unit 226(n). That is, in this example, coordinate transformation and pixel interpolation are performed in the same thread. However, the exemplary embodiment is not limited to this. Specifically, some exemplary embodiments may be configured to perform coordinate transformation for a certain strip and pixel interpolation of a transformed image obtained by this coordinate transformation in different threads. In addition, the number of threads performing coordinate transformation and the number of threads performing pixel interpolation may be the same or different.

[0126] 7 shows an example of processing executed by the coordinate conversion units 225(0) to 225(N-1) and pixel interpolation units 226(0) to 226(N-1). For each n=0, 1, . . . , N-1, the coordinate conversion unit 225(n) converts a strip 300(n) defined in the αβ coordinate system into a strip 301(n) defined in the xy coordinate system.

[0127] As mentioned above, the coordinate transformation (remapping) performed by coordinate transformation unit 225(n) is output-dependent, and different pixels in the αβ coordinate system may be transformed to the same pixel in the xy coordinate system. Thus, for each n=0, 1, . . . , N-1, coordinate transformation unit 225(n) is configured to perform the transformation from strip 300(n) to strip 301(n) as a pixel-by-pixel sequential process.

[0128] Meanwhile, the N coordinate conversion units 225(0) to 225(N-1) all perform in parallel the conversion from strip 300(0) to strip 301(0), the conversion from strip 300(1) to strip 301(1), ..., and the conversion from strip 300(N-1) to strip 301(N-1). In other words, the N coordinate conversion units 225(0) to 225(N-1) perform the coordinate conversion of the N strips 300(n) as parallel processing.

[0129] As mentioned above, the number of coordinate conversion units 225(n) does not have to be equal to N (the number of strips 300(n)) and may be, for example, less than N. However, by executing the coordinate conversion of N strips 300(n) as task parallel processing of N coordinate conversion units 225(n), it is possible to perform the entire coordinate conversion processing at the highest speed.

[0130] As shown in Fig. 7, the strip 301(n) obtained by the coordinate transformation has "missing dots" due to the influence of output dependency, etc. In other words, some pixels in the strip 301(n) have no value (brightness value). Therefore, it is necessary to apply pixel interpolation to the strip 301(n).

[0131] For each n=0, 1, . . . , N-1, pixel interpolation unit 226(n) applies pixel interpolation to strip 301(n) defined in the xy coordinate system. Pixel interpolation unit 226(n) assigns values ​​derived from the values ​​of surrounding pixels to pixels in strip 301(n) that were not assigned values ​​in the coordinate transformation of strip 300(n) defined in the αβ coordinate system.

[0132] As mentioned above, pixel interpolation can be output-dependent, so for each n=0, 1, ..., N-1, pixel interpolator 226(n) is configured to perform pixel interpolation for strip 301(n) as a pixel-by-pixel sequential process.

[0133] Meanwhile, the N pixel interpolation units 226(0) to 226(N-1) as a whole perform pixel interpolation of the strip 301(0), pixel interpolation of the strip 301(1), ..., and pixel interpolation of the strip 301(N-1) in parallel. In other words, the N pixel interpolation units 226(0) to 226(N-1) perform pixel interpolation of the N strips 301(n) as parallel processing.

[0134] As mentioned above, the number of pixel interpolation units 226(n) does not have to be equal to N (the number of strips 301(n)) and may be, for example, less than N. However, by executing pixel interpolation of N strips 301(n) as task parallel processing of N pixel interpolation units 226(n), it is possible to perform the entire pixel interpolation process at the highest speed.

[0135] In this way, N strips 302(0) to 302(N-1) defined in a real space coordinate system (xy coordinate system) are obtained after pixel interpolation. By applying registration and merging processes to these strips 302(0) to 302(N-1), an image in which motion artifacts have been corrected is obtained. These processes are described below.

[0136] As a characteristic of the Lissajous scan, any two of the N strips 302(0) to 302(N-1) have an overlapping area. Similar to the method described in Non-Patent Document 1, the image construction unit 220 is configured to perform registration between the strips by utilizing the overlap between the strips.

[0137] The image construction unit 220 first orders the N strips 302(0) to 302(N-1) according to size (area, etc.) (the first to Nth strips) and designates the first strip, which is the largest strip, as the initial reference strip. Next, the image construction unit 220 registers the second strip with respect to the first strip as a reference, and merges (combines) the first strip and the second strip.

[0138] The image constructing unit 220 registers the third strip using the merged strip obtained as a reference, and merges the merged strip with the third strip. By sequentially performing such registration and merging processes in the above order, the first to Nth strips are aligned and pasted together, and an image in which motion artifacts have been corrected is obtained.

[0139] In this registration, a cross-correlation function is used to determine the relative position between a reference strip and other strips, but since the strips can have arbitrary shapes, the correlation between the strips is calculated using a mask that treats each strip as an image of a specific shape (e.g., square) (see Appendix A of Non-Patent Document 1). This registration corresponds to "rough lateral motion correction" in Non-Patent Document 1 (page 1787).

[0140] Furthermore, the image construction unit 220 may be configured to perform registration equivalent to "fine lateral motion correction" (page 1789) in Non-Patent Document 1 to remove small lateral motion artifacts caused by eye movements such as slow drift and tremor.

[0141] In addition, the image construction unit 220 may be configured to perform registration equivalent to "axial motion correction (rough axial motion correction and / or fine axial motion correction)" in Non-Patent Document 1 (pages 1789 to 1790) to remove motion artifacts in the z direction (axial direction).

[0142] Through the above processing, three-dimensional image data of the subject's eye E in which motion artifacts have been corrected is obtained.

[0143] The image construction unit 220 is realized by cooperation between hardware including a processor and image construction software.

[0144] Data Processing Unit 230 The data processing unit 230 includes a processor and applies various types of data processing to the image of the subject's eye E. For example, the data processing unit 230 is realized by cooperation between hardware including a processor and data processing software.

[0145] The data processing unit 230 may be configured to perform registration between two images acquired of the fundus Ef. For example, the data processing unit 230 is configured to perform registration between three-dimensional image data acquired by OCT (for example, three-dimensional image data obtained by a Lissajous scan) and a front image acquired by the fundus camera unit 2. The data processing unit 230 is also configured to perform registration between two OCT images acquired by OCT. The data processing unit 230 is also configured to perform registration between two front images acquired by the fundus camera unit 2. The data processing unit 230 may also be configured to apply registration to the analysis results of the OCT images and the analysis results of the front images. These registrations can be performed by known methods, and include, for example, feature point extraction and affine transformation.

[0146] <User Interface 240> The user interface 240 includes a display unit 241 and an operation unit 242. The display unit 241 includes the display device 3. The operation unit 242 includes various operation devices and input devices. The user interface 240 may include a device such as a touch panel that integrates a display function and an operation function. It is also possible to construct an embodiment that does not include at least a part of the user interface 240. For example, the display device may be an external device connected to the ophthalmologic apparatus 1.

[0147] <Operation> The operation of the ophthalmic apparatus 1 will be described. An example of the operation of the ophthalmic apparatus 1 is shown in Fig. 8. It is assumed that preparatory processes similar to those in the past, such as input of a patient ID, setting of a scanning mode (designation of Lissajous scan), presentation of a fixation target, alignment, focus adjustment, and OCT optical path length adjustment, have already been performed.

[0148] (S1: Lissajous scan) Upon receiving a predetermined scan start trigger signal, the scan control unit 2111 starts applying an OCT scan (Lissajous scan) to the subject's eye E (fundus Ef).

[0149] The scan start trigger signal is generated, for example, in response to completion of a predetermined preparatory operation (alignment, focus adjustment, OCT optical path length adjustment, etc.) or in response to a scan start instruction operation being performed using the operation unit 242.

[0150] The scan control unit 2111 applies a Lissajous scan to a three-dimensional region of the subject's eye E by controlling the optical scanner 44, the OCT unit 100, etc. based on a scan protocol 2121 (a protocol corresponding to the Lissajous scan). Data collected by the Lissajous scan is sent to the image construction unit 220.

[0151] (S2: FFT (pre-processing, post-processing)) The FFT section 221 of the image constructing section 220 constructs three-dimensional image data (volume) by applying a fast Fourier transform to the data collected in step S1. Here, the FFT section 221 may perform predetermined pre-processing and / or post-processing.

[0152] (S3: Building multiple strips) The strip construction unit 222 divides the volume constructed in step 2 into a plurality of sub-volumes in which relatively large movements do not occur, and constructs a front projection image of each sub-volume. Each front projection image is a strip defined in the Lissajous coordinate system. As a result, it is assumed that N strips 300(0) to 300(N-1) shown in FIG. 7 are obtained.

[0153] (S4: Setting up multiple threads) The image allocation unit 223 sets threads in the image processing unit 224 based on the number of strips obtained in step S3, for example. The image allocation unit 223 sets threads whose number is equal to or less than the number of strips obtained in step S3.

[0154] In this example, the image allocation unit 223 sets threads whose number is equal to the number of strips obtained in step S3. That is, as shown in FIG. 4B, N threads 224(0) to 224(N-1) are set in the image processing unit 224.

[0155] (S5: Assign multiple strips to multiple threads) The image allocation unit 223 allocates the N strips 300(0) to 300(N-1) constructed in step 3 to the N threads 224(0) to 224(N-1) set in step S4. In this example, for each n=0, 1, . . . , N-1, strips 300(n) are allocated to threads 224(n).

[0156] (S6: Coordinate transformation (parallel processing)) N coordinate transformation units 225(0) to 225(N-1) in N threads 224(0) to 224(N-1) apply coordinate transformation to N strips 300(0) to 300(N-1), respectively. These N coordinate transformations are executed in parallel. As a result, N strips 301(0) to 301(N-1) shown in FIG. 7 are obtained. Each strip 301(n) is defined in a real space coordinate system.

[0157] In this step, the N coordinate transformations are performed in parallel as a whole, but the coordinate transformation unit 225(n) of each thread 224(n) performs the coordinate transformation of multiple pixels in the strip 300(n) as a sequential process.

[0158] For each n=0, 1, . . . , N-1, the strip 301(n) generated by the coordinate transformation unit 225(n) is input to a pixel interpolation unit 226(n) in the same thread 224(n).

[0159] (S7: Pixel Interpolation (Parallel Processing)) N pixel interpolation units 226(0) to 226(N-1) in N threads 224(0) to 224(N-1) apply pixel interpolation to N strips 301(0) to 301(N-1), respectively. These N pixel interpolations are executed in parallel. As a result, N strips 302(0) to 302(N-1) shown in FIG. 7 are obtained. Each strip 302(n) is defined in a real space coordinate system, and the above-mentioned "dropped dots" are also interpolated.

[0160] In this step, the N pixel interpolations are performed in parallel as a whole, but the pixel interpolation unit 226(n) of each thread 224(n) sequentially performs the interpolation of multiple pixels in the strip 301(n) that correspond to the “dropped dots.”

[0161] (S8: Registration and Merging (Iterative Processing)) The image constructing unit 220 orders the N strips 302(0) to 302(N-1) constructed in step S7 according to their dimensions (area, etc.), and repeatedly performs registration and merging of the reference strips and the target strips in the manner described above (i.e., according to the process described in Non-Patent Document 1). As a result, a merged image of the N strips 302(0) to 302(N-1) is constructed.

[0162] (S9: Save the final merged image) The merged image finally obtained by the repeated processing of step S8 is an image in which motion artifacts have been corrected and which represents the entire range of application of the Lissajous scan of step S1. The main controller 211 stores this final merged image in the memory 212 (and / or other storage device).

[0163] The image constructing section 220 or the data processing section 230 can use the result of the registration based on the N strips 302(0) to 302(N-1) to perform registration of the data (three-dimensional data) collected in step S1 and / or the three-dimensional image data constructed by the image constructing section 220 from the three-dimensional data. This registration includes, for example, a process of converting the defined coordinate system (αβγ coordinate system) of the Lissajous scan into a three-dimensional orthogonal coordinate system (xyz coordinate system). The main control section 211 can store the three-dimensional image data thus obtained in the storage section 212 (and / or another storage device) together with or instead of the final merged image (end).

[0164] The main control unit 211 can display the final merged image on the display unit 241 (and / or other display device). Furthermore, the image construction unit 220 or the data processing unit 230 can create an arbitrary rendering image from the three-dimensional image data constructed by the above coordinate conversion. The main control unit 211 can display this rendering image on the display unit 241 (and / or other display device).

[0165] <Action, effect, etc.> Some features of the exemplary embodiments will be described, and some operations and advantages thereof will be described.

[0166] The ophthalmic device 1 according to the above exemplary embodiment provides some aspects of a scanning imaging device, in addition to some aspects of a method for controlling a scanning imaging device, some aspects of an image processing device, some aspects of a method for controlling an image processing device, some aspects of an image processing method, and some aspects of a scanning imaging method.

[0167] It is possible to apply any of the matters described in some aspects of the method for controlling a scanning imaging device, some aspects of the image processing device, some aspects of the method for controlling an image processing device, some aspects of the image processing method, and some aspects of the scanning imaging method to the ophthalmic device 1 according to the above exemplary embodiment (more generally, any aspect of the scanning imaging device).

[0168] Now, the scanning imaging device (ophthalmologic apparatus 1) according to the exemplary embodiment includes a scanning unit, an image constructing unit (an FFT unit 221 and a strip constructing unit 222), an image processing unit (224), and an image allocation unit (223). The image processing unit (224) includes coordinate conversion units (225(0) to 225(N-1)).

[0169] The scanning unit is configured to collect data by applying a scan to a sample (the subject's eye E, the fundus Ef). This scan is performed according to a two-dimensional pattern including a series of cycles. This scan may be, for example, a scan using light (optical scan).

[0170] In some exemplary embodiments, the scanning unit may include a deflector (optical scanner 44) capable of deflecting light in a first direction (x direction) and a second direction (y direction) that are different from each other, and may be configured to apply a scan to the sample that follows a two-dimensional pattern by repeatedly changing the deflection direction along the first direction with a first period while repeatedly changing the deflection direction along the second direction with a second period that is different from the first period.

[0171] In some exemplary aspects, the deflector (optical scanner 44) may be configured to be controlled based on a preset scanning protocol (2121) based on the Lissajous function. In other words, the scan applied to the sample may be performed according to a two-dimensional pattern based on a preset scanning protocol based on the Lissajous function. An example of such an optical scan is the Lissajous scan in the above exemplary embodiment.

[0172] In some exemplary embodiments, any two of the series of cycles included in the two-dimensional pattern of scanning may cross each other.

[0173] In some exemplary embodiments, the scanner may be configured to apply an OCT scan to the sample, or may be configured to function as an SLO, perform other optical scanning, or perform other types of scanning besides optical scanning.

[0174] In the above exemplary embodiment, the scanning section includes the OCT unit 100 and elements within the fundus camera unit 2 that constitute the measurement arm (retroreflector 41, OCT focusing lens 43, optical scanner 44, objective lens 22, etc.), and applies an OCT scan to the test eye E to collect OCT data.

[0175] The image constructing unit is configured to construct a plurality of images from the data collected by the scanning unit. In the above embodiment, the FFT unit 221 and the strip constructing unit 222 construct N strips 300(0) to 300(N-1).

[0176] The image processing unit (224) is configured to be capable of multi-thread processing, and typically includes a GPU. This GPU is used as a GPGPU. In the above embodiment, N threads 224(0) to 224(N-1) are provided in the image processing unit 224.

[0177] The image allocation unit (223) allocates a plurality of images constructed by the image construction unit to a plurality of threads in the image processing unit (224). In the above exemplary embodiment, the image allocation unit 223 allocates N strips 300(0) to 300(N-1) constructed by the FFT unit 221 and the strip construction unit 222 to N threads 224(0) to 224(N-1) set in the image processing unit 224.

[0178] In the above exemplary embodiment, the number of threads is equal to the number of strips, but as mentioned above, this is not limited to this. In the above exemplary embodiment, the correspondence between strips and threads is one-to-one, but more generally, the correspondence between strips and threads may be many-to-one.

[0179] Coordinate conversion units (225(0) to 225(N-1)) provided in image processing unit 224 are configured to execute coordinate conversion of a plurality of images as at least partial parallel processing in a plurality of threads (224(0) to 224(N-1)).

[0180] In the above exemplary embodiment, N coordinate conversion units 225(0) to 225(N-1) are distributed among N threads 224(0) to 224(N-1). In other words, the ophthalmologic apparatus 1 is configured such that all of the N coordinate conversion units 225(0) to 225(N-1) function as a "coordinate conversion unit."

[0181] According to such an exemplary aspect, it is possible to perform coordinate transformation of a plurality of images constructed from data collected by applying a scan (e.g., a Lissajous scan) according to a two-dimensional pattern including a series of cycles to a sample, as at least partial parallel processing by an image processing unit capable of multi-thread processing. Therefore, it is possible to perform processing in a shorter time than when image transformation of a plurality of images is performed as sequential processing. For example, in the above exemplary embodiment, coordinate transformation of N strips 300(0) to 300(N-1) is performed by N threads 224(0) to 224(N-1), so the time required for all coordinate transformation is 1 / N compared to the case of sequential processing. Thus, according to the exemplary aspect, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan. It is possible to combine any of the matters described in the ophthalmologic apparatus 1 according to the above exemplary embodiment with the scanning imaging apparatus according to the exemplary aspect.

[0182] In some example aspects, the coordinate transformation unit may be configured to perform coordinate transformations of a plurality of pixels as a sequential process for each of a plurality of images constructed by the image construction unit.

[0183] For example, in the above exemplary embodiment, each coordinate transformer 225(n) is configured to perform the transformation from strip 300(n) to strip 301(n) as a pixel-by-pixel sequential process.

[0184] With this configuration, it is possible to execute the coordinate transformation of a plurality of images (at least partially) as parallel processing, while executing the coordinate transformation of each image as sequential processing for the plurality of pixels that compose it. Therefore, it is possible to speed up the coordinate transformation and to perform the coordinate transformation of each image suitably without being affected by output dependency, etc.

[0185] In some exemplary embodiments, the number of threads set in the image processing unit may be equal to the number of images constructed by the image construction unit, in which case the coordinate transformation unit can execute, in each of the threads, the coordinate transformation of a plurality of pixels in one of the plurality of images as a sequential process.

[0186] For example, in the above exemplary embodiment, the number (N) of the multiple threads 224(0) to 224(N-1) set in the image processing unit 224 is equal to the number (N) of the multiple strips 300(0) to 300(N-1) constructed by the FFT unit 221 and the strip construction unit 222. In addition, each coordinate transformation unit 225(n) executes coordinate transformation of multiple pixels in one strip 300(n) as sequential processing in the thread 224(n) including it.

[0187] With this configuration, it is possible to execute all of the coordinate transformations of the multiple images in parallel, while executing the coordinate transformation of each image as a sequential process for the multiple pixels that compose it, thereby maximizing the speed of the coordinate transformation and enabling the coordinate transformation of each image to be performed suitably without being affected by output dependency, etc.

[0188] In some exemplary embodiments, the scanning unit may be configured to collect data by applying a scan to a three-dimensional region of the sample. Furthermore, the image construction unit may be configured to construct a series of two-dimensional images corresponding to a series of cycles included in the two-dimensional scanning pattern from the data collected from the three-dimensional region, and construct a first set of frontal images as a plurality of images by applying a depth projection to the series of two-dimensional images. In addition, the coordinate transformation unit may be configured to transform the first set of frontal images into a second set of frontal images defined in a real space two-dimensional Cartesian coordinate system orthogonal to the depth direction.

[0189] For example, in the above exemplary embodiment, the ophthalmologic apparatus 1 is configured to collect data by applying an OCT scan (Lissajous scan) to a three-dimensional region of the subject's eye E. The FFT unit 221 and the strip construction unit 222 are configured to construct a series of two-dimensional images corresponding to a series of cycles included in the Lissajous scan pattern from the data collected from the three-dimensional region of the subject's eye E. As a result, for each cycle, a two-dimensional image representing a cross section extending in a direction along the cycle and in a depth direction (γ direction, z direction) is obtained. Furthermore, the strip construction unit 222 is configured to construct N strips 300(0) to 300(N-1) defined in a Lissajous coordinate system (αβ coordinate system) by applying a projection in the depth direction (γ direction, z direction) (and the process described in Non-Patent Document 1) to the series of two-dimensional images. The coordinate conversion units 225(0) to 225(N-1) are configured to convert the N strips 300(0) to 300(N-1) into N strips 301(0) to 301(N-1) defined in a real space two-dimensional Cartesian coordinate system (xy coordinate system) that is orthogonal to the depth direction (γ direction, z direction).

[0190] According to this configuration, it is possible to provide a specific configuration for speeding up coordinate transformation.

[0191] The parallelized processing is not limited to coordinate transformation. In some exemplary embodiments, the image processing unit may include a pixel interpolation unit. The pixel interpolation unit may be configured to perform pixel interpolation of a plurality of images (a plurality of transformed images) obtained from a plurality of images by the coordinate transformation unit as at least a partial parallel processing. Furthermore, for each of the plurality of images constructed by the image construction unit, the image processing unit may be configured to perform the coordinate transformation and the pixel interpolation in the same thread.

[0192] For example, in the above exemplary embodiment, pixel interpolation units 226(0)-226(N-1) provided in the image processing unit 224 perform pixel interpolation of N strips 301(0)-301(N-1) obtained from N strips 300(0)-300(N-1) by the coordinate conversion units 225(0)-225(N-1) as parallel processing in multiple threads 224(0)-224(N-1).

[0193] In the above exemplary embodiment, N pixel interpolation units 226(0) to 226(N-1) are distributed among N threads 224(0) to 224(N-1). In other words, the ophthalmologic apparatus 1 is configured such that all of the N pixel interpolation units 226(0) to 226(N-1) function as a "pixel interpolation unit."

[0194] According to this configuration, in scanning imaging such as a Lissajous scan, it is possible to speed up not only coordinate conversion but also pixel interpolation.

[0195] It should be noted that processes that can be parallelized are not limited to coordinate transformation and pixel interpolation, and any step in the process of creating image data in which motion artifacts have been corrected from data collected by a Lissajous scan or the like can be parallelized.

[0196] An exemplary embodiment of a method for controlling a scanning imaging device that can be provided by the ophthalmic apparatus 1 according to the exemplary embodiment described above will be described. In some exemplary embodiments, the scanning imaging device includes a scanning unit that applies a scan to a sample and collects data, and a processor, and a method for controlling the scanning imaging device may include the steps described below.

[0197] A method for controlling a scanning imaging apparatus includes controlling a scanner to collect data by applying a scan to a sample according to a two-dimensional pattern including a series of cycles, and controlling a processor to construct a plurality of images from the data collected by the scanner, assign the constructed images to a plurality of threads, and perform coordinate transformation of the images in the plurality of threads as at least partial parallel processing.

[0198] According to such a control method for a scanning imaging device, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan, as with the scanning imaging device (e.g., the ophthalmic device 1) according to the exemplary embodiment. Any of the matters described in the ophthalmic device 1 according to the above exemplary embodiment can be combined with the control method for a scanning imaging device according to the exemplary embodiment.

[0199] An exemplary embodiment of an image processing device that can be provided by the ophthalmologic apparatus 1 according to the exemplary embodiment will be described. In some exemplary embodiments, the image processing device may include a storage unit, an image construction unit, an image processing unit, and an image allocation unit. Furthermore, the image processing unit may include a coordinate conversion unit.

[0200] The memory unit is configured to store data collected by applying a scan to the sample according to a two-dimensional pattern including a series of cycles, and corresponds to memory unit 212 in the above exemplary embodiment.

[0201] The image constructor is configured to construct a number of images from the data stored in the memory unit. In the above exemplary embodiment, this image constructor corresponds to the FFT unit 221 and the strip constructor 222.

[0202] The image processing unit is configured to be capable of multi-thread processing. In the above exemplary embodiment, this image processing unit corresponds to the image processing unit 224.

[0203] The image allocator is configured to allocate a number of images to a number of threads of the image processing unit. In the above exemplary embodiment, this image allocator corresponds to the image allocator 223.

[0204] The image processing unit includes a coordinate conversion unit that performs coordinate conversion of a plurality of images in a plurality of threads as at least partially parallel processing. In the above exemplary embodiment, this coordinate conversion unit corresponds to the coordinate conversion units 225(0) to 225(N-1).

[0205] According to such an image processing device, like the scanning imaging device (e.g., the ophthalmic device 1) according to the exemplary embodiment, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan. Any of the items described in the ophthalmic device 1 according to the above exemplary embodiment can be combined with the image processing device according to the exemplary embodiment.

[0206] An exemplary embodiment of a control method for an image processing device that can be provided by the ophthalmologic apparatus 1 according to the exemplary embodiment described above will be described. In some exemplary embodiments, the image processing device includes a storage unit and a processor, and the method for controlling the image processing device may include the steps described below.

[0207] A method for controlling an image processing device includes: applying a scan to a sample according to a two-dimensional pattern including a series of cycles to store collected data in a memory unit; and controlling a processor to construct a plurality of images from the data stored in the memory unit, assign the constructed plurality of images to a plurality of threads, and perform coordinate transformation of the plurality of images in the plurality of threads as at least partially parallel processing.

[0208] According to such a control method for an image processing device, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan, as with a scanning imaging device (e.g., the ophthalmic device 1) according to an exemplary embodiment. Any of the matters described in the ophthalmic device 1 according to the above exemplary embodiment can be combined with the control method for an image processing device according to an exemplary embodiment.

[0209] An exemplary embodiment of a scanning imaging method that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the scanning imaging method may include the steps described below.

[0210] First, the scanning imaging method applies a scan according to a two-dimensional pattern including a series of cycles to a sample to collect data, constructs a plurality of images from the collected data, and further assigns the constructed plurality of images to a plurality of threads, and executes coordinate transformation of the plurality of images in the threads as at least partial parallel processing.

[0211] According to such a scanning imaging method, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan, as with the scanning imaging device (e.g., the ophthalmic device 1) according to the exemplary embodiment. Any of the matters described in the ophthalmic device 1 according to the above exemplary embodiment can be combined with the scanning imaging method according to the exemplary embodiment.

[0212] An exemplary embodiment of an image processing method that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the image processing method may include the steps described below.

[0213] First, the image processing method prepares data collected by applying a scan to a sample according to a two-dimensional pattern including a series of cycles. For example, the data of the sample is received via a communication line or a recording medium. Alternatively, the data is collected by applying a scan to the sample.

[0214] The image processing method further comprises constructing a plurality of images from the prepared data, allocating the constructed plurality of images to a plurality of threads, and performing coordinate transformation of the plurality of images in the plurality of threads as at least partially parallel processing.

[0215] In some exemplary embodiments, the image processing method may prepare a plurality of images constructed from data collected by applying a scan to a sample according to a two-dimensional pattern including a series of cycles. For example, the plurality of images may be received via a communication line or a recording medium, or the plurality of images may be constructed from data collected by applying a scan to a sample. In this case, the image processing method assigns the prepared plurality of images to a plurality of threads, and performs coordinate transformation of the plurality of images in the plurality of threads as at least partial parallel processing.

[0216] According to such an image processing method, it is possible to speed up data processing (at least coordinate transformation) in scanning imaging such as a Lissajous scan, as with the scanning imaging device (e.g., the ophthalmic device 1) according to the exemplary embodiment. Any of the matters described in the ophthalmic device 1 according to the above exemplary embodiment can be combined with the image processing method according to the exemplary embodiment.

[0217] In some exemplary aspects, it is possible to provide a program for causing a computer to execute any aspect of the method for controlling a scanning imaging device, a program for causing a computer to execute any aspect of the method for controlling an image processing device, a program for causing a computer to execute any aspect of the image processing method, or a program for causing a computer to execute any aspect of the scanning imaging method.

[0218] It is also possible to create a computer-readable non-transitory recording medium having such a program recorded therein. This non-transitory recording medium may be in any form, examples of which include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0219] The present disclosure merely illustrates some aspects and is not intended to limit the invention. Those who intend to implement the present invention can make any modifications (omissions, substitutions, additions, etc.) within the scope of the gist of the present invention. [Explanation of symbols]

[0220] 1 Ophthalmology equipment 44 Optical Scanner 100 OCT units 211 Main control unit 2111 Scanning control section 212 Storage section 2121 Scanning Protocol 220 Image Construction Department 211 Fast Fourier Transform (FFT) Section 222 Strip Construction Section 223 Image Allocation Section 224 Image Processing Unit 224(0)~224(N-1), 224(n) threads 225(0)~225(N-1), 225(n) Coordinate conversion section 226(0)~226(N-1), 226(n) Pixel Interpolation Section 300(0)~300(N-1), 300(n) strip (Lissajous coordinate system) 301(0)~301(N-1), 301(n) Strip (real space coordinate system) 302(0)~302(N-1), 302(n) Strip (pixel interpolated)

Claims

1. a scanning unit for collecting data by applying a scan to the sample according to a two-dimensional pattern including a series of cycles; an image constructing unit that constructs a plurality of mutually different images from the data; an image processing unit capable of multi-thread processing; an image allocation unit that allocates the plurality of images to a plurality of threads of the image processing unit; Including, the image processing unit includes a coordinate conversion unit that executes coordinate conversion of the plurality of images assigned to the plurality of threads, respectively, as at least partial parallel processing; Scanning imaging device.

2. For each of the plurality of images, the coordinate transformation unit executes coordinate transformation of a plurality of pixels as a sequential process.

2. The scanning imaging device of claim 1.

3. the number of the plurality of threads is equal to the number of the plurality of images; the coordinate transformation unit sequentially performs, in each of the plurality of threads, coordinate transformation of a plurality of pixels in one image assigned to the thread from among the plurality of images; 3. The scanning imaging device of claim 2.

4. the scanning component collects the data by applying the scan to a three-dimensional region of the sample; The image constructing unit includes: constructing from said data a series of two-dimensional images corresponding to said series of cycles; and constructing a first set of frontal images of the plurality of images by applying a depth projection to the series of two-dimensional images; the coordinate conversion unit converts the first front image group into a second front image group defined in a real space two-dimensional orthogonal coordinate system orthogonal to the depth direction. The scanning imaging device according to any one of claims 1 to 3.

5. The image processing unit further includes a pixel interpolation unit that executes pixel interpolation of a plurality of transformed images obtained from the plurality of images by the coordinate transformation as at least partial parallel processing. The scanning imaging device according to any one of claims 1 to 4.

6. For each of the plurality of images, the image processing unit executes the coordinate transformation and the pixel interpolation in the same thread.

6. The scanning imaging device of claim 5.

7. The scanning unit is a deflector capable of deflecting light in a first direction and a second direction different from each other; and applying the scan to the sample by repeatedly varying the deflection direction along the first direction with a first period while repeatedly varying the deflection direction along the second direction with a second period different from the first period; The scanning imaging device according to any one of claims 1 to 6.

8. The deflector is controlled based on a preset scanning protocol based on a Lissajous function.

8. The scanning imaging device of claim 7.

9. The series of cycles intersect with each other.

9. A scanning imaging device according to claim 7 or 8.

10. The scanning unit applies an optical coherence tomography (OCT) scan to the sample. The scanning imaging apparatus according to any one of claims 1 to 9.

11. 1. A method for controlling a scanning imaging device including a scanner for applying a scan to a sample to collect data, and a processor, comprising: controlling the scanner to scan the sample according to a two-dimensional pattern including a series of cycles to collect data; The processor, constructing a plurality of distinct images from said data; assigning the plurality of images to a plurality of threads, respectively; Executing the coordinate transformations of the plurality of images assigned to the plurality of threads as at least partially parallel processes. Control so that A method for controlling a scanning imaging device.

12. a memory unit for storing data collected by scanning the sample according to a two-dimensional pattern including a series of cycles; an image constructing unit that constructs a plurality of mutually different images from the data; an image processing unit capable of multi-thread processing; an image allocation unit that allocates the plurality of images to a plurality of threads of the image processing unit; Including, the image processing unit includes a coordinate conversion unit that executes coordinate conversion of the plurality of images assigned to the plurality of threads, respectively, as at least partial parallel processing; Image processing device.

13. A method for controlling an image processing device including a memory unit and a processor, comprising: applying a scan to the sample according to a two-dimensional pattern including a series of cycles and storing collected data in the memory unit; The processor, constructing a plurality of distinct images from said data; assigning the plurality of images to a plurality of threads, respectively; Executing the coordinate transformations of the plurality of images assigned to the plurality of threads as at least partially parallel processes. Control so that A method for controlling an image processing device.

14. collecting data by applying a scan to the sample according to a two-dimensional pattern comprising a series of cycles; constructing a plurality of distinct images from said data; assigning the plurality of images to a plurality of threads, respectively; executing the coordinate transformations of the plurality of images respectively assigned to the plurality of threads as at least partial parallel processing; Scanning imaging methods.

15. preparing collected data by applying a scan to the sample according to a two-dimensional pattern comprising a series of cycles; constructing a plurality of distinct images from said data; assigning the plurality of images to a plurality of threads, respectively; executing the coordinate transformations of the plurality of images respectively assigned to the plurality of threads as at least partial parallel processing; Image processing methods.

16. A program for causing a computer to execute the method according to any one of claims 11, 13, 14 and 15.

17. A non-transitory computer-readable recording medium having the program of claim 16 recorded thereon.

Images