Ophthalmologic apparatus

Abstract

To provide an ophthalmologic apparatus capable of quickly confirming, through real-time processing, whether a subject's face is moving during alignment or while acquiring ocular information.SOLUTION: An ophthalmologic apparatus A includes: an optical system 50 that acquires ocular information on a subject's eye E to be examined; a face support section (chin rest section 30 and forehead rest section 33) that supports the subject's face; and a control section 60 that controls each section of the apparatus. At a position for capturing an anterior segment of the eye E to be examined, an event camera 80 is mounted which outputs only event data in which brightness change in each pixel data from frames of the anterior-segment image of the eye E to be examined exceeds a set threshold in combination of coordinates and time information. The control section 60 includes a facial dynamic state monitoring processing part 64 that extracts an inner corner point IC and an outer corner point OC of the eye E to be examined using output information from the event camera 80 and monitors a facial movement of the subject on the basis of the inner corner point IC and the outer corner point OC.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an ophthalmic apparatus. [Background technology]

[0002] Patent Document 1 proposes an ophthalmologic apparatus equipped with a determination unit that determines whether the face is properly supported by a face support unit including a chin rest based on the result of detection of a pupil image for each anterior eye image by a pupil image detection unit, in order to properly support the face of the subject by the face support unit. Patent Document 2 proposes an eye tracking method and apparatus that is applied to a head-worn device and uses event camera data, including determining the user's eye tracking characteristics based on light intensity data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-154105 [Patent Document 2] Patent No. 7008131 Summary of the Invention [Problem to be solved by the invention]

[0004] Ophthalmic devices perform alignment to adjust the positional relationship between the optical system and the subject's eye, subjective and objective measurements of the subject's eye characteristics, and capture images of the subject's eye, such as fundus images, with the subject's chin resting on a chin rest. As a general rule, the face of the subject, whose chin is resting on the chin rest, must remain stationary during alignment and ocular information acquisition. Therefore, there is a demand for ophthalmic devices that can confirm in real time whether the subject's face, supported by the chin rest, is remaining stationary or moving during alignment and ocular information acquisition. Note that "acquiring ocular information" refers to acquiring ocular characteristic values ​​through subjective measurement of the subject's eye characteristics, acquiring ocular characteristic values ​​through objective measurement of the subject's eye characteristics, and acquiring images of the subject's eye by capturing fundus images, etc.

[0005] In contrast, the technology described in Patent Document 1 uses anterior eye image data acquired from an anterior eye camera and determines that the face is properly supported if the following conditions are met for a certain period of time: the amount of facial movement is within a threshold value and the rotation angle is within a range corresponding to fixational eye movement. Therefore, the technology described in Patent Document 1 requires a process for detecting the position and shape of the pupil image, a process for detecting the position and rotation angle of the subject's eye, a process for determining whether the detected facial movement amount is a condition, a process for determining whether the detected rotation angle is a condition, and a process for determining whether the duration is a condition. Therefore, the technology described in Patent Document 1 first checks the state of the subject's face and the pupil of the subject's eye, then performs alignment, and then proceeds to acquiring eye information. Therefore, even if the subject's face moves during alignment or eye information acquisition, it is not possible to detect this.

[0006] The technology described in Patent Document 2 is an eye tracking technology that uses an event camera to detect the movement of a person's pupils and track their gaze. Therefore, even if the technology described in Patent Document 2 is applied to an ophthalmic device, it is not possible to check whether the subject's face is moving during alignment or while acquiring eye information. As a result, when performing alignment, if the subject's face moves during alignment, it may take a long time to complete the adjustment of the positional relationship between the optical system and the subject's eye, or the adjustment may fail. Furthermore, when measuring eye characteristics as eye information of the subject's eye, if the subject's face moves during measurement, the accuracy of the acquired eye characteristics may decrease or the acquisition of eye characteristics may fail. Furthermore, when capturing eye information such as a fundus image, if the subject's face moves during capture, blurring or flare may occur in the fundus image, or the capture itself may fail.

[0007] The present invention has been made in response to the above-mentioned problems, and aims to provide an ophthalmic device that can quickly check, through real-time processing, whether the subject's face is moving during alignment or while acquiring eye information. [Means for solving the problem]

[0008] In order to solve the above problems, the ophthalmologic apparatus of the present invention includes an optical system for acquiring ocular information of a subject's eye to be examined, a face support unit for supporting the face of the subject, and a control unit for controlling each unit of the apparatus. An event camera is attached at a position capturing the anterior segment of the subject's eye, and the event camera outputs only event data in which a luminance change in each pixel data of an anterior segment image frame of the subject's eye exceeds a set threshold, in combination with coordinate and time information. The control unit includes a facial movement monitoring processing unit that uses the output information from the event camera to extract the inner corner and outer corner of the eye of the subject's eye, and monitors the facial movement of the subject based on the inner corner and outer corner of the eye. [Effects of the Invention]

[0009] The ophthalmic device of the present invention monitors the movement of the subject's face based on the inner and outer corners of the eyes extracted using output information from an event camera, and can quickly confirm through real-time processing whether the subject's face is moving during alignment or while eye information is being acquired. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an overall perspective view showing an ophthalmic system including an ophthalmic apparatus according to a first embodiment. [Figure 2] 1 is a front view showing the configuration of the ophthalmologic apparatus of the first embodiment as viewed from the chin rest side. [Figure 3] 1 is a side view showing an ophthalmologic apparatus according to a first embodiment. [Figure 4] FIG. 2 is a block diagram showing a control system configuration in the ophthalmologic apparatus of the first embodiment. [Figure 5] FIG. 10 is a flowchart of the photographing process for an image of an eye to be examined in conjunction with face / pupil dynamics monitoring. [Figure 6] FIG. 10 is an operational flow diagram showing a face movement monitoring process operation and an eye movement monitoring process operation. [Figure 7] FIG. 2 is a diagram showing an example of an anterior eye image acquired by an anterior eye camera. [Figure 8] FIG. 10 is a diagram illustrating an example of event data acquired by an event camera. [Figure 9A] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face moves upward. [Figure 9B] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face moves sideways. [Figure 10A] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face moves forward. [Figure 10B] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face moves backward. [Figure 11A] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face is tilted to the left. [Figure 11B] FIG. 10 is an explanatory diagram showing a first determination example when the subject's face is tilted to the right. [Figure 12] FIG. 10 is an explanatory diagram showing face direction detection using event cameras in a stereo arrangement. [Figure 13A] FIG. 10 is an explanatory diagram showing a second determination example when the upper eyelid is closed by blinking. [Figure 13B] FIG. 10 is an explanatory diagram showing a second determination example when the subject's eye has ptosis. [Figure 14A] FIG. 10 is an explanatory diagram showing a second determination example when the gaze is directed from the front to the left. [Figure 14B] FIG. 10 is an explanatory diagram showing a second determination example when the gaze is directed from the front to the right. DETAILED DESCRIPTION OF THE INVENTION

[0011] An ophthalmic apparatus according to an embodiment of the present invention will be described below based on a first embodiment shown in the drawings.

[0012] The ophthalmologic apparatus applied to Example 1 is an apparatus that observes, photographs, and records an anterior segment image of the eye to be examined, an eye fundus image of the eye to be examined, and a tomographic image of the eye fundus of the eye to be examined, and provides them as electronic images for diagnosis. Note that, in the figure, X, Y, and Z indicate the X-axis, the left-right axis in the left-right direction (horizontal direction) when facing the main body of the ophthalmologic apparatus with the eye to be examined as a reference, the Y-axis, the up-down axis in the up-down direction (vertical direction), and the Z-axis, the front-back axis in the front-back direction (depth direction) perpendicular to the X-axis and Y-axis. [Example]

[0013] [Overall configuration of the device (Fig. 1 to 3)] As shown in FIG. 1, the ophthalmologic device A includes a stand unit 10, a main body unit 20, a chin rest unit 30, a control panel unit 40, an optical system 50, a control unit 60, an optical table 70, and an event camera 80.

[0014] The ophthalmic device A includes a fundus camera that captures fundus images of the subject's eye E and an OCT (abbreviation for "Optical Coherence Tomography") that captures fundus tomographic images of the subject's eye E, and is called a 3D fundus imaging device. Here, the fundus camera is a camera that images the fundus condition of the retina, optic nerve, capillaries, and other components at the back of the subject's eye E and captures the fundus image. The OCT is an optical coherence tomography that uses light interference to image the tomographic layers of the retina present at the fundus of the subject's eye E and captures the fundus tomographic images.

[0015] The pedestal unit 10 is placed on an optical table 70 whose height can be adjusted in the Y-axis direction. The pedestal unit 10 supports the main body unit 20 on its upper surface so that it can move in three axial directions: the X-axis, the Y-axis, and the Z-axis. The pedestal unit 10 has a chin rest 30 on its front surface. The pedestal unit 10 has a power switch 11, a power inlet 12, a USB terminal 13, and a LAN terminal 14 on both side surfaces. "USB" is an abbreviation for "Universal Serial Bus," and "LAN" is an abbreviation for "Local Area Network." The USB terminal 13 is a terminal for connecting external memory, and as shown in FIG. 3, an HDD (abbreviation for "Hard Disk Drive"), a USB memory, etc., is connected to the LAN terminal 14 via a LAN cable 15.

[0016] 3, the gantry 10 has a built-in power supply unit 17 and an XYZ drive unit 18. The power supply unit 17 includes a power switch 11, a power inlet 12, a USB terminal 13, a LAN terminal 14, etc. The XYZ drive unit 18 is a motor actuator having motors and motor drive circuits that drive the main body 20 in each of the three axial directions (three-dimensional directions) of the X-axis, Y-axis, and Z-axis when moving the main body 20 relative to the gantry 10 during alignment control.

[0017] The main body 20 is provided so as to be movable in the X-axis, Y-axis, and Z-axis directions by an XYZ drive unit 18 relative to the stand 10 to which the chin rest 30 is fixed. The main body 20 has an optical system 50 built in a main body cover 21 that covers the entire main body 20, which acquires ocular information of the subject's eye E with the chin J supported against the chin rest 30. As shown in FIGS. 1 and 3, a control panel unit 40 is disposed at an upper position on the back of the main body cover 21. In addition to the optical system 50, a control unit 60 is built in the internal space of the main body cover 21, as shown in FIG. 3.

[0018] 2, the main body cover 21 has, at the center of the front surface, an objective lens 51 of the optical system 50 that faces the subject's eye E. The objective lens 51 has, at its peripheral positions, an anterior eye camera 22, an event camera 80, a peripheral fixation lamp 23, and an anterior eye observation filter 24.

[0019] The anterior eye camera 22 is a camera that captures an anterior eye image by photographing the anterior eye of the subject. This anterior eye camera 22 is composed of two cameras, a right anterior eye camera 22a and a left anterior eye camera 22b, that are stereo-arranged on either side of the objective lens 51 with their lens optical axes tilted toward the anterior eye of the subject's eye E, which is the subject of the image capture. The right anterior eye camera 22a and the left anterior eye camera 22b capture right and left anterior eye images that are cropped from a portion of the subject's face supported by the chin rest 30, depending on the selection of the subject's eye E and the angle of view at that time. Furthermore, the right anterior eye camera 22a and the left anterior eye camera 22b are positioned with their width dimensions in the X-axis direction and their angle of view determined, so that the three-dimensional coordinate position of the subject's eye E can be identified by calculation processing based on the two anterior eye images.

[0020] The peripheral fixation lights 23 are fixation lights used to fix the line of sight of the subject's eye E by turning them on, and eight of them are arranged at equal intervals around the periphery of the objective lens 51. The anterior eye observation filters 24 are filters used to adjust the amount of light during anterior eye observation or anterior eye OCT, and two filters are arranged vertically outside the right anterior eye camera 22a and two filters are arranged vertically outside the left anterior eye camera 22b (four filters in total). The event camera 80 will be described in detail later.

[0021] The face support unit of Example 1 is composed of a chin rest 30 and a forehead rest 33. The chin rest 30 is a member that supports the subject's chin J, and as shown in Figures 2 and 3, is provided so as to be movable up and down relative to the chin rest support 31 and the forehead rest 33 that are fixed to the pedestal unit 10. The chin rest 30 has an elevation rod 30a that moves up and down by a built-in chin rest drive unit 32, a chin rest 30b fixed at the upper end of the elevation rod 30a, and chin rest stop pins 30c provided on both sides of the chin rest 30b.

[0022] As shown in Fig. 2, the chin rest support unit 31 has forehead rests 33 fixed to both ends of the T-shape, each shaped to surround the subject's face on three sides with the subject's chin J supported on the chin rest 30. The forehead rest 33 is fixed to the pedestal 10 and has a pair of vertical support columns 331, 332 extending in the Y-axis direction and a horizontal support column 333 connecting the upper ends of the pair of vertical support columns 331, 332. The pair of vertical support columns 331, 332 are provided with eye height lines 33a, which serve as a guide for adjusting the subject's eye E to an appropriate height position, at positions that correspond to the center height of the measurement unit opening defined by the objective lens 51. The horizontal support column 333 has a detachable forehead rest 33b, made of an elastic material such as silicone rubber, attached to the center of the surface that comes into contact with the subject's forehead F. Furthermore, the horizontal support portion 333 has an arm 34 that can be bent in multiple stages at the center position of the surface opposite to the surface that contacts the subject's forehead F, and an external fixation target (not shown) is provided at the tip of the arm 34.

[0023] As shown in Figs. 1 and 3, the control panel unit 40 is disposed at an upper position on the back surface of the main body cover 21. The control panel unit 40 has a display screen 41 that displays in color an anterior ocular segment image of the subject's eye E from the anterior ocular segment camera 22 and an anterior ocular segment observation image of the subject's eye E from the optical system 50. The display screen 41 is a touch panel that allows the examiner to input operations to the control unit 60 by touching displayed button images and images with his / her finger. The control panel unit 40 is connected and supported to the main body unit 20 by a connecting support unit 42. The connecting support unit 42 has a combined support structure of bending support and rotation support that allows the display screen 41 to be set at any position in the entire circumferential direction relative to the main body unit 20 and also allows the tilt angle of the display screen 41 to be freely set.

[0024] The control panel unit 40 is used when the examiner examines the eye characteristics while standing close to the subject. The connection support unit 42 ensures the function of allowing the examiner to position the display screen 41 at a position that is easy for the examiner to operate, regardless of where the examiner is located around the ophthalmologic apparatus A. The remote control tablet 40' shown in FIG. 1 is used when the examiner examines the eye characteristics by remote control. The remote control tablet 40' has a touch panel display screen 41' and has input operation functions equivalent to those of the control panel unit 40, as well as a communication function with the main body unit 20.

[0025] The optical system 50 acquires ocular information of the subject's eye E while the subject supports his or her chin on the chin rest 30, and as shown in Fig. 3, includes a fundus camera unit 52 having an objective lens 51, and an OCT unit 53. The fundus camera unit 52 includes an illumination optical system and a photographing optical system, and is a unit constituting a fundus camera that acquires a fundus image of the subject's eye E using a lens, an image sensor, etc. The OCT unit 53 is a unit constituting an OCT that acquires a fundus tomographic image of the subject's eye E using a wavelength-tunable light source, a fiber coupler, etc. Note that the optical system 50 has the function of acquiring a fundus image and a fundus tomographic image of the subject's eye E, as well as the function of acquiring an observation image of the anterior segment of the subject's eye E.

[0026] The control unit 60 controls each unit of the device (fundus camera unit 52, OCT unit 53, chin rest unit 30, main body unit 20, etc.) based on various input operations including touch operations on the display screen 41 of the control panel unit 40. As shown in Fig. 3, the control unit 60 has, as its hardware configuration, a control board 60a, a CPU board 60b, and an image board 60c. The control unit 60 will be described in detail later.

[0027] As shown in FIG. 1 , the optical table 70 includes a tabletop 71 on which the ophthalmic apparatus A is placed, tabletop supports 72, caster supports 73, a table lifting mechanism 74, a lifting lever 75, a caster base 76, and casters 77. The table lifting mechanism 74 is disposed inside the tabletop supports 72 and the caster supports 73, and raises and lowers the tabletop supports 72 relative to the caster supports 73. The lifting lever 75 is a manual operation unit that manually raises and lowers the tabletop 71 and the tabletop supports 72 on which the ophthalmic apparatus A is placed. The tabletop stroke amount is set to, for example, about 200 mm. Here, the optical table 70 of Example 1 has a height suitable for imaging in a seated position because the ophthalmic apparatus A is used for performing measurements while the subject is seated (in a seated position) on a chair 78 with an adjustable seat height. The optical table can also be an ophthalmic device that takes images while the subject is standing (standing position), in which case there is no need to provide a chair 78, and a table with a height that corresponds to standing position measurements is used.

[0028] The event camera 80 is a camera positioned to capture the anterior segment of the subject's eye E. The camera combines coordinate and time information to output only event data from pixel data in an anterior segment image frame of the subject's eye E, where the change in luminance exceeds a set threshold. The event data includes positive events (bright areas) output when a comparator detects an increase in luminance exceeding a positive threshold, and negative events (dark areas) output when a comparator detects a decrease in luminance exceeding a negative threshold. The event camera 80 is composed of two stereo cameras, a right event camera 80a and a left event camera 80b, positioned on either side of the objective lens 51, with their optical axes tilted toward the anterior segment of the subject's eye E, which is the subject of imaging. The right event camera 80a is positioned above the right anterior segment camera 22a, and the left event camera 80b is positioned above the left anterior segment camera 22b. The right event camera 80a and the left event camera 80b are positioned with their width W in the X-axis direction and their angle of view α determined (see FIG. 12).

[0029] [Control system configuration (Fig. 4)] The control system of the ophthalmologic apparatus A includes a control panel unit 40 (display screen 41), an optical system 50 (fundus camera unit 52, OCT unit 53), and a control unit 60, as shown in FIG.

[0030] The control unit 60 includes a main control unit 61 that controls the fundus camera unit 52 and the OCT unit 53, a memory unit 62 that stores necessary data, an alignment control unit 63, a facial movement monitoring processing unit 64, and an eye movement monitoring processing unit 65.

[0031] The alignment control unit 63 performs alignment control to adjust the positional relationship between the optical system 50 and the subject's eye E when the subject sits in front of the device and places his or her chin on the chin rest 30. The alignment control unit 63 inputs anterior eye images from the stereoscopically arranged anterior eye cameras 22 (right anterior eye camera 22a, left anterior eye camera 22b) and outputs drive commands to the XYZ drive unit 18 or the chin rest drive unit 32. The chin rest drive unit 32 is used to move the subject's face in the Y-axis direction in chin rest height control. The XYZ drive unit 18 is used to adjust movement in three axes in auto alignment and manual alignment. As shown in FIG. 4, the alignment control unit 63 has a chin rest height control unit 631, an auto alignment unit 632, and a manual alignment unit 633.

[0032] The chin rest height control unit 631 controls the output of a drive command to the chin rest drive unit 32 to align the position of the subject's eye E with the height of the eye height line 33a while the subject's chin J is placed on the chin rest unit 30 and the subject's forehead F is pressed against the forehead rest unit 33b.

[0033] The auto-alignment unit 632 performs auto-alignment of the pupil based on the anterior eye image acquired by the anterior eye camera 22. The auto-alignment of the pupil is performed by automatically adjusting the photographed eye so that the pupil marks of the two anterior eye images displayed in the video area of ​​the display screen 41 are aligned. The automatic adjustment of the photographed eye is a control that automatically adjusts the XYZ axis alignment by driving the XYZ drive unit 18 so that the two pupil marks are positioned so as to overlap at the center position of the anterior eye image.

[0034] The manual alignment unit 633 performs manual alignment of the pupil when a preset time limit has elapsed since the start of auto-alignment, or when the examiner actively selects manual operation. For manual alignment of the pupil, tapping the manual mode button displayed on the auto-alignment screen stops the automatic adjustment of the photographed eye and switches to manual adjustment mode, in which the photographed eye is manually adjusted. The manual adjustment mode is performed by manually tapping the pupil marks on the two anterior eye images displayed on the display screen 41. When the examiner taps, the XYZ drive unit 18 drives and performs XYZ axis alignment adjustment so that the two pupil marks are positioned overlapping at the center position of the anterior eye image.

[0035] The facial movement monitoring processor 64 extracts the inner corner and outer corner of the subject's eye E using output information from the event camera 80, and monitors the facial movement of the subject based on the inner corner and outer corner of the eye. The facial movement monitoring processor 64 has a first extraction unit 641 that extracts the inner corner and outer corner of the eye of the subject's eye using output information from the event camera 80, and a first determination unit 642 that determines the facial movement of the subject based on the extraction results of the inner corner and outer corner of the eye. In addition to determining the facial movement of the subject, the first determination unit 642 determines the direction of the subject's face based on the extraction results of the inner corner and outer corner of the eye obtained by the stereo-arranged right event camera 80a and left event camera 80b.

[0036] The eye movement monitoring processing unit 65 uses output information from the event camera 80 to extract a pupil circle representing the pupil shape of the eye E and a bright spot projected onto the cornea of ​​the eye E from the optical system 50, and monitors the movement of the eye E based on the pupil circle and the bright spot. The eye movement monitoring processing unit 65 has a second extraction unit 651 that uses output information from the event camera 80 to extract the pupil circle and the bright spot of the eye E, and a second determination unit 652 that determines the movement of the eye E based on the extraction results of the pupil circle and the bright spot.

[0037] Here, the facial movement monitoring processing unit 64 and the eye movement monitoring processing unit 65 are preferably configured using an FPGA (abbreviation for Field Programmable Gate Array) that can be adapted to the event camera 80 in edge computing technology that processes data by the ophthalmic apparatus A. Note that an FPGA is a device that can instantly correct any errors in logic circuit design on the spot using hardware language, and when the event camera 80 is applied to the ophthalmic apparatus A, it can be easily adapted to monitor / determine the facial movements of the subject and the movements of the subject's eye E.

[0038] [Photography and processing of images of the subject's eye while monitoring facial and eye movements (Fig. 5)] The photographing process for photographing an image of the subject's eye (for example, an anterior eye image, a fundus image, or a tomographic image of the fundus) using facial / ocular dynamics monitoring executed by the control unit 60 will be described below with reference to Fig. 5. The photographing process starts when the subject is identified by the ophthalmologic apparatus A with the power switch turned on.

[0039] In step S1, the examiner registers the patient (= subject) by entering the name and patient ID to identify the patient. Here, the "patient ID" is an identification number for managing the subject's personal information related to the ophthalmic examination, and can include information such as age, gender, and past examination information for follow-up.

[0040] Step S2 is a step in which the examiner selects the type of imaging. This selection of imaging type is performed by touching and selecting one of the imaging modes (anterior eye image imaging mode, fundus image imaging mode, fundus tomographic image imaging mode, etc.) from the imaging icon selection screen displayed on the display screen 41 of the control panel unit 40. In this step S2, the type of imaging is selected and the eye to be photographed is selected by touching the display button.

[0041] Step S3 is a step in which the chin rest height control unit 631 of the alignment control unit 63 controls the chin rest height. This step S3 starts after the height of the top plate 71 of the optical table 70 or the chair 78 is adjusted so that the subject can rest their chin J on the chin rest unit 30 in a comfortable position, the examiner is instructed to place the subject's chin J on the chin rest unit 30, and the chin J is placed. In step S3, the examiner presses the chin rest up / down movement button displayed on the display screen 41 of the control panel unit 40 to adjust the height of the chin rest unit 30 so that the position of the outer corner of the eye E to be examined is aligned with the height position of the eye height line 33a on the vertical support units 331, 332 of the chin rest unit 30. After the chin rest height control is completed in step S3, the examiner instructs the subject to press their forehead F against the forehead support unit 33b.

[0042] Step S4 is a step in which the auto-alignment unit 632 of the alignment control unit 63 executes auto-alignment (automatic adjustment) for the pupil of the photographed eye. Note that if the auto-alignment time is prolonged or if the examiner actively desires it, the manual alignment unit 633 executes manual alignment (manual adjustment) for the pupil instead of auto-alignment (automatic adjustment) for the pupil.

[0043] Step S5 is a step in which the main control unit 61 performs autofocus, which automatically adjusts the focus. This autofocus adjusts the focus to the anterior segment of the subject's eye E in the anterior segment image capturing mode, and adjusts the focus to the fundus of the subject's eye E in the fundus image capturing mode and fundus tomographic image capturing mode.

[0044] Step S6 is a step in which the examiner captures an image of the eye to be examined (for example, an anterior segment image, a fundus image, or a tomographic image of the fundus). This capture is performed by tapping the "OK button" on the capture screen displayed on the display screen 41, and the examiner moves on to the next capture. A preview of the captured image is displayed after each capture.

[0045] In step S7, the examiner checks the preview of the captured image and determines whether the preview is OK or NG. If it is determined in step S7 that the preview is OK, the process proceeds to step S8. If it is determined in step S7 that the preview is NG, the process returns to step S5, autofocus is performed again, and photographing is attempted in step S6 until it is determined in step S7 that the preview is OK.

[0046] Step S8 is a step in which the examiner saves the captured image in the storage unit 62. When the examiner taps the "save button" on the shooting screen displayed on the display screen 41, the captured image at that time is saved in the storage unit 62, and when saving is complete, the process proceeds to END.

[0047] Step S9 is a step in which the face movement monitoring processing unit 64 and eye movement monitoring processing unit 65 of the control unit 60 read a predetermined number of pieces of output information (event data, data position coordinates, time) from the event camera 80. Here, the predetermined number of pieces of output information refers to the number of pieces of output information acquired within a determined determination time for determining the subject's facial movement and eye movement based on the output information from the event camera 80. Note that the process flow, which repeatedly proceeds from step S9 to step S10 to step S11, is executed in parallel with the process flow from step S4 onwards (auto-alignment, auto-focus, photographing).

[0048] In step S10, the face movement monitoring processing unit 64 and the eye movement monitoring processing unit 65 of the control unit 60 perform face movement monitoring processing and eye movement monitoring processing based on a predetermined number of pieces of output information read from the event camera 80. The detailed processing operations of the face movement monitoring processing and eye movement monitoring processing will be described later.

[0049] Step S11 is a step in which the face movement monitoring processing unit 64 and eye movement monitoring processing unit 65 of the control unit 60 determine whether or not the storage of the captured images in the memory unit 62 is complete. If step S11 determines that the storage of the captured images is not complete, the process returns to step S9, a predetermined number of pieces of output information are newly read, and the face movement monitoring process and eye movement monitoring process are repeated. If step S11 determines that the storage of the captured images is complete, the process proceeds to END.

[0050] [Facial movement monitoring processing operation and eye movement monitoring processing operation (Fig. 6)] The facial movement monitoring processing operation and eye movement monitoring processing operation executed in step S10 of Fig. 5 will be described below with reference to Fig. 6. Note that the eye E to be examined has the inner corner of the eye IC, the outer corner of the eye OC, the upper eyelid line UL, the lower eyelid line LL, the sclera (white of the eye) WE, the cornea (black of the eye) IR, the pupil circle EP, the bright spot BS, the distance between the inner corner of the eye IC and the outer corner of the eye OC L, and the center of the eye E to be examined CP (see Fig. 7).

[0051] In step S101, the first extraction unit 641 of the face movement monitoring processing unit 64 extracts the inner corner point IC and outer corner point OC of the subject's eye E. In step S101, the output information from the event camera 80 is used to extract the upper eyelid line UL and the lower eyelid line LL, which change in luminance along the line segments of the upper and lower eyelids due to the difference in luminance between the eyelid and the sclera (white of the eye) WE of the subject's eye E when the subject's face moves. In step S101, of the two intersections of the upper eyelid line UL and the lower eyelid line LL, the inner intersection is extracted as the inner corner point IC of the subject's eye E, and the outer intersection is extracted as the outer corner point OC.

[0052] Step S102 is a step in which the first determination unit 642 of the face movement monitoring processing unit 64 determines that the subject's face has been moved away from the face support units (chin rest 30, forehead rest 33). This step S102 monitors the movements of the extracted eye inner corner point IC and eye outer corner point OC, and when the eye inner corner point IC and eye outer corner point OC are moving in the same upward direction on a two-dimensional coordinate plane defined by the X and Y axes, it determines that the subject's chin J has been moved away from the chin rest 30 and the face is facing upward. Step S102 monitors the movements of the extracted eye inner corner point IC and eye outer corner point OC, and when the eye inner corner point IC and eye outer corner point OC are moving in the same horizontal direction on a two-dimensional coordinate plane defined by the X and Y axes, it determines that the subject's face has been moved away from the chin rest 30 and the forehead rest 33 and the face is facing sideways.

[0053] Step S103 is a step in which, when the first determination unit 642 of the facial movement monitoring processing unit 64 determines that the subject's chin J is separated, the determination result is output as feedback information. This feedback information is output by outputting, for example, a command to display, on the display screen 41 of the control panel unit 40, "The subject's chin is separated from the chin rest." The feedback information may be output as a command to announce the determination result via the built-in speaker, or by outputting either or both of these commands, the examiner is made aware that the subject's chin J is separated. Furthermore, in addition to outputting a command to make the examiner aware of the situation, the feedback information may also be output as a command to temporarily suspend the alignment process, imaging process, or the like currently being executed. When a temporary suspension command is output, a command to cancel the suspension and resume the process is output when it is determined that the subject's chin J is placed in the appropriate position on the chin rest 30.

[0054] Step S104 is a step in which the first determination unit 642 of the face movement monitoring processing unit 64 determines that the subject's face is moving in the Z-axis direction. This step S104 monitors changes in the distance L between the extracted eye inner corner point IC and eye outer corner point OC, and determines that an increase in distance L indicates that the subject's face is moving toward the subject in the Z-axis direction, and that a decrease in distance L indicates that the subject's face is moving away from the subject in the Z-axis direction.

[0055] Step S105 is a step in which, when the first determination unit 642 of the facial movement monitoring processing unit 64 determines that the subject's face is moving in the Z-axis direction, the determination result is output as feedback information. This feedback information is output by outputting a command to display, for example, "The subject's face is approaching the device" or "The subject's face is moving away from the device" on the display screen 41 of the control panel unit 40. The feedback information may be output as a command to announce the determination result through a built-in speaker, or one or both of these commands may be output to make the examiner aware that the subject's face is moving in the Z-axis direction. Furthermore, the feedback information may be output as a command to temporarily suspend an alignment process, an imaging process, or the like, in addition to a command to make the examiner aware of the movement, as in step S103.

[0056] Step S106 is a step in which the first determination unit 642 of the face movement monitoring processing unit 64 determines that the subject's face is tilted. This step S106 monitors the movement of the extracted eye inner corner point IC and eye outer corner point OC relative to the center point CP of the subject's eye E, and if the eye inner corner point IC and eye outer corner point OC are rotating relative to the center point CP, it is determined that the subject's face is tilted in the rotating direction.

[0057] Step S107 is a step in which, when the first determination unit 642 of the facial movement monitoring processing unit 64 determines that the subject's face is tilted, the determination result is output as feedback information. This feedback information is output by outputting a command to display, for example, "The subject's face is tilted to the right" or "The subject's face is tilted to the left" on the display screen 41 of the control panel unit 40. The feedback information may be output as a command to announce the determination result through a built-in speaker, or one or both of these commands may be output to make the examiner aware that the subject's face is tilted. Furthermore, the feedback information may be output as a command to temporarily suspend an alignment process, an imaging process, or the like, in addition to a command to make the examiner aware of the tilt, as in step S103.

[0058] Step S108 is a step in which the first determination unit 642 of the face movement monitoring processing unit 64 determines the direction of the subject's face. This step S106 calculates the three-dimensional coordinate positions of the eye inner corner point IC and the eye outer corner point OC based on the extraction results of the eye inner corner point IC and the eye outer corner point OC by the stereoscopically arranged right event camera 80a and left event camera 80b, and determines that the direction perpendicular to the line connecting the eye inner corner point IC and the eye outer corner point OC in the three-dimensional coordinate space is the direction of the face.

[0059] Step S109 is a step in which, when the first determination unit 642 of the facial movement monitoring processing unit 64 determines the orientation of the subject's face, it outputs the determination result as feedback information. This feedback information is output by outputting a command to display, for example, "The subject's face is facing forward" or "The subject's face is facing diagonally to the right" on the display screen 41 of the control panel unit 40. The feedback information may also be output as a command to announce the determination result via a built-in speaker. Either or both of these command outputs may be used to prompt the examiner to recognize the orientation of the subject's face. Note that, because the orientation of the subject's face may be photographed or measured at an angle that is not facing forward, no command to suspend the alignment process or the photographing process is output, and the examiner's judgment is left to the examiner's judgment after recognizing the orientation of the subject's face.

[0060] In step S111, the second extraction unit 651 of the ocular dynamics monitoring processing unit 65 extracts a pupil circle EP representing the pupil shape of the subject's eye E and a bright spot BS projected from the optical system 50 onto the cornea IR of the subject's eye E using output information from the event camera 80. Here, the bright spot BS is an alignment bright spot formed by a parallel beam of light projected from the alignment system of the optical system 50 onto the cornea IR of the subject's eye E. The pupil circle EP is extracted as a contour region whose luminance changes due to the difference in luminance with the cornea (iris) IR when the position of the pupil of the subject's eye E changes at minimum luminance, using output information from the event camera 80. The bright spot BS is extracted as a point region whose luminance changes due to a change in position in accordance with the change in luminance caused by the change in pupil position when the direction of the subject's eye's gaze changes. Note that the amount of change in position of the bright spot BS does not match the amount of change in position of the pupil and is smaller than the amount of change in position of the pupil.

[0061] Step S112 is a step in which the second determination unit 652 determines whether the subject's eye E is blinking or has ptosis. This step S112 monitors the shapes of the extracted pupil circle EP and bright spot BS, and determines that the subject's eye E is blinking when the pupil circle EP and bright spot BS have disappeared, and determines that the subject's eye E has ptosis when the pupil circle EP remains missing.

[0062] Step S113 is a step in which, when the second determination unit 652 of the ocular dynamics monitoring processing unit 65 determines blinking or ptosis of the subject's eye E, the determination result is output as feedback information. This feedback information is output by outputting a command to display, for example, "Blinking" or "Eyelids are half-open" on the display screen 41 of the control panel unit 40. The feedback information may also be output as a command to announce the determination result through a built-in speaker. Either or both of these command outputs allow the examiner to recognize that the subject's eye E is blinking or that its eyelids are half-open. Note that in these cases, the problem can be resolved by instructions or assistance from the examiner, such as "Please refrain from blinking for a while," and therefore no command to temporarily suspend alignment processing, imaging processing, or the like is output.

[0063] Step S114 is a step in which the second determination unit 652 determines the line of sight of the subject's eye E. This step S114 monitors positional changes of the extracted pupil circle EP and bright spot BS, and determines the line of sight of the subject's eye E based on the positional changes of the pupil circle EP and bright spot BS. In particular, step S114 determines the line of sight of the subject's eye E based on the direction of change in the position of the pupil circle EP as the direction of change in the line of sight.

[0064] Step S115 is a step in which, when the second determination unit 652 of the ocular dynamics monitoring processing unit 65 determines the line of sight of the subject's eye E, the second determination unit 652 outputs the determination result as feedback information. This feedback information is output by outputting a command to display, for example, "The line of sight is facing forward" or "The line of sight has shifted sideways" on the display screen 41 of the control panel unit 40. The feedback information may also be output as a command to announce the determination result through a built-in speaker, and either or both of these command outputs allow the examiner to recognize the line of sight of the subject's eye E. Note that in these cases, the problem can be solved by the examiner's instruction, such as "Please change the line of sight to the front," so no command to temporarily suspend the alignment process, the imaging process, or the like is output.

[0065] [Comparison of anterior segment camera images and event camera images (Figures 7 and 8)] The image from the anterior eye camera 22 becomes the anterior eye image FI shown in FIG. 7. When the upper eyelid line UL and the lower eyelid line LL are drawn for the subject's eye E in this anterior eye image FI, the innermost of the two intersections is the inner corner point IC, and the innermost of the two intersections is the outer corner point OC. The internal region of the upper eyelid line UL and the lower eyelid line LL includes the cornea IR, which is the iris of the eye, the sclera WE, which are the white of the eye on both sides of the cornea IR, and the pupil circle EP, which is the central region of the cornea IR. The bright spot BS is projected onto the cornea IR as an alignment bright spot. For example, in the case of the subject's eye E facing forward, it is located at the center of the pupil circle EP. Note that the distance L is defined as the linear distance between the inner corner point IC and the outer corner point OC. The center point CP of the subject's eye E is defined as the center position of the line between the inner corner point IC and the outer corner point OC.

[0066] A characteristic of the pupil circle EP is that it is extremely dark. A characteristic of the scleral WE is that, in contrast to the pupil circle EP, it is extremely bright. The characteristics of a dark pupil circle EP and a bright scleral WE are common to all races. For this reason, the pupil circle EP has a clear outline due to the difference in brightness. The upper eyelid line UL and lower eyelid line LL become clear line segments due to the difference in brightness. The bright point BS becomes a clear point due to the difference in brightness.

[0067] For example, when the subject's face moves upward, the anterior segment image frame of the subject's eye E captured by the event camera 80 becomes an event data image EI as shown in FIG. 8 . When a contour, line segment, or point with a difference in brightness moves, the event data image EI obtains positive event data (white circles in FIG. 8 ) if the brightness change in the brighter direction exceeds a set threshold, and negative event data (black circles in FIG. 8 ) if the brightness change in the darker direction exceeds a set threshold. In other words, the event data image EI is obtained independently and asynchronously from the event camera 80 for each pixel as dynamic information representing movement only when there is movement in a portion (contour, line segment, point, etc.) with a brightness difference compared to its surroundings. Here, setting the brightness gain of the sclera WE to a value around 255 as the threshold for positive event data makes it easier to output positive event data along the line segments of the upper eyelid line UL and the lower eyelid line LL. Setting the brightness gain of the pupil circle EP to a value around 0 as the threshold for negative event data makes it easier to output negative event data along the contour of the pupil circle EP.

[0068] For this reason, the amount of data in the event data image EI is much smaller than the amount of data in the anterior eye image FI, which outputs the brightness data of all pixels. Incidentally, the frame rate of the event camera 80 exceeds 1000 FPS, whereas that of a normal anterior eye camera 22 is about 60 FPS. The FPS, which indicates the frame rate, is a unit (short for "frames per second") that indicates how many images a video consists of in one second.

[0069] In this way, the output information from the event camera 80 consists of event data information consisting of positive event data and negative event data for each pixel, data position coordinate information on a two-dimensional XY coordinate plane, and time information when the data was acquired. Therefore, using the output information from the event camera 80 (event data, data position coordinate, time), it is possible to perform high-speed processing to extract moving parts, determine facial behavior, and determine eye behavior in real time during alignment or while capturing an image of the subject's eye.

[0070] Here, the facial movement determination process determines facial movement by analyzing the movement of the extracted eye inner corner point IC and eye outer corner point OC over time. This is because the eye inner corner point IC and eye outer corner point OC move in accordance with the subject's facial movement, and it is possible to grasp how the face is moving from the relative movement of these two distant positions. The eye movement determination process determines eye movement by analyzing the movement of the extracted pupil circle EP and bright point BS over time. This is because the pupil circle EP and bright point BS can grasp the opening and closing movement of the upper eyelid by monitoring changes in their shape, and pupil movement can be grasped by monitoring their position.

[0071] [Facial movement monitoring processing (Fig. 6, Fig. 9 to Fig. 12)] The facial movement monitoring process is performed in the order of step S101, step S102, step S103, step S104, step S105, step S106, step S107, step S108, and step S109 in the flowchart of FIG.

[0072] In step S101, the inner corner point IC and outer corner point OC of the subject's eye E are extracted using output information from the event camera 80. In the first stage of the extraction method, the upper eyelid line UL and lower eyelid line LL, which change in brightness along the line segments of the upper and lower eyelids due to the difference in brightness between the eyelids and the sclera WE of the subject's eye E when the subject's face moves, are extracted. In the second stage, of the two intersections of the upper eyelid line UL and lower eyelid line LL, the inner intersection is extracted as the inner corner point IC of the subject's eye E, and the outer intersection is extracted as the outer corner point OC. Facial movement is determined in step S102 and thereafter by monitoring the movements of the extracted inner corner point IC and outer corner point OC.

[0073] (Jaw separation assessment of subject) In step S102, as shown in the explanatory diagram of Fig. 9A, it is assumed that the subject moves from the positions of the eye inner corner point IC' and the eye outer corner point OC' to the positions of the eye inner corner point IC and the eye outer corner point OC along the arrows on the XY coordinate plane. At this time, the first determination unit 642 obtains a monitoring result that the eye inner corner point IC and the eye outer corner point OC move in the same upward direction by the same amount, and thereby determines that the subject's chin J has been released from the chin rest 30 and the face is facing upward.

[0074] In step S102, as shown in the explanatory diagram of Fig. 9B, it is assumed that the subject moves from the positions of the eye inner corner point IC' and the eye outer corner point OC' to the positions of the eye inner corner point IC and the eye outer corner point OC along the arrows on the XY coordinate plane. At this time, the first determination unit 642 obtains a monitoring result that the eye inner corner point IC and the eye outer corner point OC move in the same horizontal direction by the same amount, and thereby determines that the subject's face has moved away from the chin rest 30 and the forehead rest 33 and is turned sideways to the left.

[0075] In step S103, when it is determined in step S102 that the subject's chin J is apart, the determination result is output as feedback information to inform the examiner that the subject's chin J is apart. Here, the determination that the subject's chin J is apart can also be made by determining that the face is facing sideways to the right, diagonally upward to the right, or diagonally upward to the left, if the chins move in the same direction and by the same amount. Furthermore, as is clear from FIGS. 9A and 9B , when it is determined that the subject's chin J is apart, the pupil circle EP and the bright spot BS may also move in the same direction and by the same amount, and the determination may be made by adding monitoring of the pupil circle EP and the bright spot BS to the inner corner of the eye IC and the outer corner of the eye OC. Note that, to make it easier to understand the movement of the inner corner of the eye IC and the outer corner of the eye OC, FIGS. 9A and 9B show an illustration of the anterior segment instead of the event data image EI represented by white and black dots as shown in FIG. 8 .

[0076] (Determining the movement of the subject's face in the Z-axis direction) 10A, it is assumed that the subject's face moves from the positions of the inner corner of the eye IC' and the outer corner of the eye OC' on the XY coordinate plane along the outward arrow to the positions of the inner corner of the eye IC and the outer corner of the eye OC. At this time, the first determination unit 642 obtains a monitoring result that the distance L between the inner corner of the eye IC and the outer corner of the eye OC has increased to the distance L', and thereby determines that the subject's face is moving closer to the main unit 20 in the Z-axis direction.

[0077] In step S104, as shown in the explanatory diagram of FIG. 10B, it is assumed that the subject's face has moved from the positions of the inner corner of the eye IC' and the outer corner of the eye OC' on the XY coordinate plane along the inward arrow to the positions of the inner corner of the eye IC and the outer corner of the eye OC. At this time, the first determination unit 642 obtains a monitoring result that the distance L between the inner corner of the eye IC and the outer corner of the eye OC has decreased to the distance L", and therefore determines that the subject's face has moved away from the main unit 20 in the Z-axis direction.

[0078] In step S105, when it is determined in step S104 that the subject's face is moving in the Z-axis direction, the determination result is output as feedback information to make the examiner aware that the subject's face is moving in the Z-axis direction. Note that, in order to make it easier to understand the movements of the eye inner corner point IC and the eye outer corner point OC, Figures 10A and 10B show an illustration image of the anterior eye segment instead of the event data image EI represented by white and black dots as shown in Figure 8.

[0079] (Judgment of subject's face tilt) 11A, it is assumed that the subject's face is moved from the positions of the inner corner of the eye IC' and the outer corner of the eye OC' on the XY coordinate plane along the right-turn arrow to the positions of the inner corner of the eye IC and the outer corner of the eye OC. At this time, the first determination unit 642 obtains a monitoring result that indicates that the inner corner of the eye IC and the outer corner of the eye OC are rotating with respect to the center CP of the subject's eye E, and thereby determines that the subject's face is tilted to the left.

[0080] 11B, it is assumed that the subject's face is moved from the positions of the inner corner point IC' and the outer corner point OC' on the XY coordinate plane along the left turning arrow to the positions of the inner corner point IC and the outer corner point OC. At this time, the first determination unit 642 obtains a monitoring result that the inner corner point IC and the outer corner point OC are moving in a turning motion with respect to the center point CP of the subject's eye E, and thereby determines that the subject's face is tilted to the right.

[0081] In step S107, when it is determined in step S106 that the subject's face is tilted, the determination result is output as feedback information to make the examiner aware that the subject's face is tilted. Note that, in order to make it easier to understand the movements of the eye inner corner point IC and the eye outer corner point OC, Figures 11A and 11B show an illustration image of the anterior eye segment instead of the event data image EI represented by white and black dots as shown in Figure 8.

[0082] (Determining the subject's face direction) In step S108, as shown in the explanatory diagram of FIG. 12, it is assumed that the right event camera 80a and the left event camera 80b capture the eye inner corner point IC and the eye outer corner point OC in a stereo arrangement. At this time, the eye inner corner point IC can be formed into a triangle based on the positional relationship between the position C1 of the right event camera 80a and the position C2 of the left event camera 80b. Similarly, the eye outer corner point OC can be formed into a triangle based on the positional relationship between the position C1 of the right event camera 80a and the position C2 of the left event camera 80b. Therefore, the coordinate value z1 of the eye inner corner point IC and the coordinate value z2 of the eye outer corner point OC are calculated using the two-dimensional coordinate values ​​(x1, y1) of the eye inner corner point IC, the two-dimensional coordinate values ​​(x2, y2) of the eye outer corner point OC, known values ​​(positions C1, C2, width W, angle of view α), and trigonometric functions. This calculation process yields the three-dimensional coordinate position (x1, y1, z1) of the eye inner corner point IC and the three-dimensional coordinate position (x2, y2, z2) of the eye outer corner point OC. In this way, once the three-dimensional coordinate positions (x1, y1, z1), (x2, y2, z2) of the inner corner of the eye IC and the outer corner of the eye OC are obtained, the first determination unit 642 determines that the direction perpendicular to the line connecting the inner corner of the eye IC and the outer corner of the eye OC in the three-dimensional coordinate space is the direction of the face.

[0083] In step S109, when the direction of the subject's face is determined in step S108, the determination result is output as feedback information to allow the examiner to recognize the direction of the subject's face. Note that Figure 12 shows an illustration of the anterior segment of the eye to make it easy to understand the positions of the inner corner point IC and the outer corner point OC, and explains the positional relationship between the right event camera 80a and the left event camera 80b.

[0084] [Effect of eye movement monitoring processing (Fig. 6, Fig. 13, Fig. 14)] The eye movement monitoring process is performed in the order of step S111, step S112, step S113, step S114, and step S115 in the flowchart of FIG.

[0085] In step S111, the pupil circle EP and bright spot BS of the subject's eye E are extracted using output information from the event camera 80. The pupil circle EP is extracted by using output information from the event camera 80 to extract a contour area whose brightness changes due to the difference in brightness with the cornea (iris) IR when the position of the pupil of the subject's eye E changes at the lowest brightness. The bright spot BS is extracted by using output information from the event camera 80 to extract a point area whose brightness changes due to position change in accordance with the change in brightness caused by the change in pupil position when the direction of the gaze of the subject's eye E changes. Ocular dynamics are determined in step S112 and onwards by monitoring the shapes and movements of the extracted pupil circle EP and bright spot BS.

[0086] (Determination of blinking and ptosis of the subject's eye) 13A, when the shapes of the extracted pupil circle EP and bright spot BS are being monitored, the pupil circle EP and bright spot BS are assumed to disappear for an instant. At this time, the second determination unit 652 obtains a monitoring result that the pupil circle EP and bright spot BS have disappeared due to the upper eyelid of the subject's eye E closing, and thereby determines that the subject's eye E is blinking.

[0087] 13B, when the shapes of the extracted pupil circle EP and bright spot BS are monitored, the pupil circle EP and bright spot BS remain missing. In this case, the second determination unit 652 obtains a monitoring result that the upper eyelid of the subject's eye E has drooped, causing a missing pupil circle EP and bright spot BS, and determines that the subject's eye E has ptosis.

[0088] In step S113, when the blinking or ptosis of the test eye E is determined in step S112, the determination result is output as feedback information to make the examiner aware that the test eye E is blinking or has ptosis. Note that Figures 13A and 13B show an illustration image of the anterior segment instead of the event data image EI represented by dots with white and black circles shown in Figure 8, in order to make it easier to understand the changes in the shape of the pupil circle EP and the bright spot BS.

[0089] (Determining the line of sight of the subject's eye) 14A, when monitoring changes in the positions of the extracted pupil circle EP and bright spot BS, the second determination unit 652 determines that the gaze has changed from straight ahead to leftward by obtaining a monitoring result indicating that the pupil circle EP and bright spot BS of the subject's eye E have changed in position.

[0090] 14B, when monitoring changes in the positions of the extracted pupil circle EP and bright spot BS, the second determination unit 652 determines that the pupil circle EP and bright spot BS have changed in position to the left in the drawing. In this case, the second determination unit 652 obtains a monitoring result indicating that the pupil circle EP and bright spot BS of the subject's eye E have changed in position, and determines that the line of sight has changed from straight ahead to rightward.

[0091] In step S115, when a change in the line of sight of the subject's eye E is determined in step S114, the change is output as feedback information to make the examiner aware of the change in the line of sight of the subject's eye E. Note that, in order to make it easier to understand the positional changes of the pupil circle EP and the bright spot BS, Figures 14A and 14B show an illustration image of the anterior segment instead of the event data image EI represented by dots made of white and black circles shown in Figure 8.

[0092] [Effects of Ophthalmic Device A] (1) The ophthalmologic apparatus A includes an optical system 50 for acquiring ocular information of the subject's eye E, a face support unit (chin rest 30, forehead rest 33) for supporting the subject's face, and a control unit 60 for controlling each unit of the apparatus. An event camera 80 is attached at a position capturing the anterior segment of the subject's eye E, and outputs only event data in which a change in luminance of each pixel in a frame of an anterior segment image of the subject's eye E exceeds a set threshold, in combination with coordinate and time information. The control unit 60 includes a facial movement monitoring processing unit 64 that uses the output information from the event camera 80 to extract the inner corner point IC and outer corner point OC of the eye E, and monitors the facial movement of the subject based on the inner corner point IC and outer corner point OC of the eye. The present invention can provide an ophthalmologic apparatus A that can quickly check whether the subject's face is moving through real-time processing during alignment or while acquiring eye information.

[0093] (2) The facial movement monitoring processing unit 64 uses output information from the event camera 80 to extract the upper eyelid line UL and lower eyelid line LL, which change in brightness along the line of the upper and lower eyelids due to the difference in brightness between the eyelids and the sclera WE (white of the eye) of the subject's eye E when the subject's face moves, and has a first extraction unit 641 that extracts the inner corner point IC and outer corner point OC of the subject's eye E from the two intersections of the upper eyelid line UL and lower eyelid line LL, and a first determination unit 642 that determines the movement of the subject's face based on the extraction results of the inner corner point IC and outer corner point OC by the first extraction unit 641. This invention can quickly determine the facial movement of the subject simply by monitoring the movement trajectories of the inner corner point IC and outer corner point OC of the subject's eye E extracted using output information from the event camera 80.

[0094] (3) The first determination unit 642 monitors the movement of the eye inner corner point IC and eye outer corner point OC extracted by the first extraction unit 641, and when the eye inner corner point IC and eye outer corner point OC are moving in the same direction on the two-dimensional coordinate plane defined by the X axis (left-right axis) and the Y axis (up-down axis), it determines that the subject's face is away from the face support unit (chin rest unit 30, forehead rest unit 33). In this invention, it can be determined that the subject's face is separated from the face support portion (chin rest portion 30, forehead rest portion 33) by the fact that the inner corner point IC of the eye and the outer corner point OC of the eye are moving in the same direction. In this invention, it can be determined that the subject's chin J is separated from the chin rest portion 30, for example.

[0095] (4) The first judgment unit 642 monitors changes in the distance L between the inner corner point IC and the outer corner point OC of the eye extracted by the first extraction unit 641, and when the distance L increases, it judges that the subject's face is moving closer in the Z-axis direction (front-to-back axis direction), and when the distance L decreases, it judges that the subject's face is moving away in the Z-axis direction. This invention can determine whether the subject's face is moving in the Z-axis direction by monitoring changes in the distance L between the inner corner of the eye IC and the outer corner of the eye OC.

[0096] (5) The first judgment unit 642 monitors the movement of the inner corner point IC and outer corner point OC of the eye extracted by the first extraction unit 641 relative to the center point CP of the subject's eye E, and if the inner corner point IC and outer corner point OC of the eye are rotating relative to the center point CP, it judges that the subject's face is tilted in the direction of the rotation. This invention can determine whether the subject's face is tilted by monitoring the rotational movement of the inner corner point IC and outer corner point OC of the eye relative to the center point CP of the inner corner point IC and outer corner point OC of the eye.

[0097] (6) The event cameras 80 are a right event camera 80a and a left event camera 80b that are stereo-positioned so as to capture the anterior segment of one eye of the subject's eye E from two different directions. The first determination unit 642 calculates three-dimensional position coordinates for each of the inner corner point IC and outer corner point OC of the eye extracted by the first extraction unit 641, and determines that the direction perpendicular to the line connecting the inner corner point IC and outer corner point OC in the three-dimensional coordinate space is the direction of the face. This invention uses a stereo-arranged right event camera 80a and left event camera 80b to capture the positions of the eye corner point IC and eye corner point OC in three-dimensional coordinate space, thereby making it possible to determine the orientation of the subject's face.

[0098] (7) The control unit 60 has an eye movement monitoring processing unit 65 that uses output information from the event camera 80 to extract a pupil circle EP representing the pupil shape of the subject's eye E and a bright spot BS projected onto the cornea of ​​the subject's eye E from the optical system 50, and monitors the movement of the subject's eye E based on the pupil circle EP and the bright spot BS. This invention makes it possible to quickly check whether there is any movement in the subject's eye E during alignment or while acquiring eye information, through real-time processing.

[0099] (8) The eye movement monitoring processing unit 65 has a second extraction unit 651 that uses output information from the event camera 80 to extract, as a pupil circle EP, a contour area whose brightness changes due to the brightness difference with the cornea IR (iris area) when the position of the pupil of the subject eye E changes, and extracts, as a bright point BS, a point area whose brightness changes when the direction of the gaze of the subject eye E changes, and a second judgment unit 652 that judges the movement of the subject eye E based on the extraction results of the pupil circle EP and bright point BS by the second extraction unit 651. According to the present invention, the movement of the subject's eye E can be quickly determined simply by monitoring changes in the shape and position of the pupil circle EP and the bright spot BS extracted using the output information from the event camera 80.

[0100] (9) The second judgment unit 652 monitors the shapes of the pupil circle EP and the bright spot BS extracted by the second extraction unit 651, and when the pupil circle EP and the bright spot BS disappear or are missing, judges that the test eye E is blinking or has ptosis. This invention can determine blinking or ptosis of the subject's eye E by monitoring changes in shape due to disappearance or loss of the pupil circle EP and bright spot BS.

[0101] (10) The second determination unit 652 monitors changes in the positions of the pupil circle EP and the bright spot BS extracted by the second extraction unit 651, and determines the line of sight of the subject's eye E based on the changes in the positions of the pupil circle EP and the bright spot BS. This invention can determine the line of sight of the subject's eye E by monitoring changes in the positions of the pupil circle EP and the bright spot BS.

[0102] The ophthalmic apparatus A of Example 1 has been described above with reference to the drawings. However, the specific configuration of the ophthalmic apparatus of the present invention is not limited to Example 1, and design changes and additions are permitted as long as they do not deviate from the gist of the invention according to each claim of the claims.

[0103] In Example 1, the face support section is shown as being composed of a chin rest 30 and a forehead rest 33. However, the face support section is not limited to Example 1 as long as it is a member that supports the subject's face to an extent that it does not move by contacting a part of the face and receiving force from the face. The face support section may, for example, be composed of only a chin rest. The face support section may also, for example, be composed of only a forehead rest. The face support section may also, for example, be composed of only a cheek rest. Furthermore, the face support section may, for example, be composed of a combination of any two or three of a chin rest, a forehead rest, and a cheek rest.

[0104] In the first embodiment, the event camera 80 includes a right event camera 80a and a left event camera 80b that are stereoscopically arranged to capture the anterior segment of one of the examinee's eyes E from two different directions. The event camera is not limited to a configuration of two stereoscopically arranged cameras. For example, the event camera may be a monocular camera that captures the anterior segment of one of the examinee's eyes from one direction.

[0105] In the first embodiment, an example is shown in which the control unit includes both a face movement monitoring processor 64 that monitors the facial movement of the subject based on the eye inner corner point IC and the eye outer corner point OC, and an eye movement monitoring processor 65 that monitors the movement of the subject's eye E based on the pupil circle EP and the bright spot BS. However, the control unit is not limited to an example in which the control unit includes both a face movement monitoring processor and an eye movement monitoring processor. For example, the control unit may include only a face movement monitoring processor that monitors the facial movement of the subject based on the eye inner corner point and the eye outer corner point.

[0106] In the first embodiment, an example of application of the present invention to an ophthalmic apparatus A is shown, which observes, photographs, and records an anterior segment image of the subject's eye, a fundus image of the subject's eye, and a tomographic image of the fundus of the subject's eye, and provides them as electronic images for diagnosis. However, the application of the present invention to an ophthalmic apparatus is not limited to an ophthalmic apparatus A equipped with an imaging optical system that photographs various images of the subject's eye. The ophthalmic apparatus to which the present invention is applied may be, for example, an ophthalmic apparatus equipped with a subjective measurement optical system that subjectively measures various ocular characteristics of the subject's eye, or an ophthalmic apparatus equipped with an objective measurement optical system that objectively measures various ocular characteristics of the subject's eye. In short, the present invention can be applied to various ophthalmic apparatuses as long as they include an optical system that acquires ocular information of the subject's eye, a chin rest that supports the subject's chin, and a control unit that controls each unit of the apparatus.

[0107] [1] an optical system that acquires eye information of a subject's eye to be examined; and a face support unit that supports the face of the subject; a control unit that controls each unit of the device, an event camera is attached at a position capturing the anterior segment of the subject's eye, and outputs only event data in which a change in brightness of each pixel data of an anterior segment image frame of the subject's eye exceeds a set threshold, in combination with coordinate and time information; and the control unit has a facial movement monitoring processing unit that uses the output information from the event camera to extract the inner corner and outer corner of the eye of the subject's eye, and monitors the facial movement of the subject based on the inner corner and outer corner of the eye. [2] [1] In the ophthalmic device described in The ophthalmologic apparatus is characterized in that the facial movement monitoring processing unit includes: a first extraction unit that uses output information from the event camera to extract upper and lower eyelid lines that change in brightness along the line segments of the upper and lower eyelids due to the difference in brightness between the eyelid and the sclera (white of the eye) of the subject's eye when the subject's face moves, and extracts the inner corner and outer corner of the eye of the subject's eye from two intersections of the upper eyelid line and the lower eyelid line; and a first determination unit that determines the movement of the subject's face based on the extraction results of the inner corner and outer corner of the eye by the first extraction unit. [3] [2] In the ophthalmic device described in The ophthalmologic device is characterized in that the first determination unit monitors the movement of the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and determines that the subject's face is away from the face support unit when the inner corner point and the outer corner point of the eye move in the same direction on a two-dimensional coordinate plane defined by a left-right axis and a top-bottom axis. [4] In the ophthalmic device described in [2] or [3], The ophthalmologic device is characterized in that the first judgment unit monitors changes in the distance between the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and when the distance increases, judges that the subject's face is moving closer in the anterior-posterior axis direction, and when the distance decreases, judges that the subject's face is moving away in the anterior-posterior axis direction. [5] In the ophthalmic device according to any one of [2] to [4], The ophthalmologic device is characterized in that the first determination unit monitors the movement of the inner corner point and the outer corner point of the eye extracted by the first extraction unit relative to the center of the subject's eye, and when the inner corner point and the outer corner point of the eye are moving in a rotating direction relative to the center, it determines that the subject's face is tilted in the rotating direction. [6] In the ophthalmic device according to any one of [2] to [5], The ophthalmologic apparatus is characterized in that the event cameras are a right event camera and a left event camera arranged in stereo so as to capture the anterior segment of one of the examinee's eyes from two different directions, and the first determination unit calculates three-dimensional position coordinates for each of the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and determines that the direction perpendicular to the line connecting the inner corner point and the outer corner point of the eye in three-dimensional coordinate space is the direction of the face. [7] In the ophthalmic device according to any one of [1] to [6], The control unit uses output information from the event camera to extract a pupil circle representing the pupil shape of the subject's eye and a bright spot projected from the optical system onto the cornea of ​​the subject's eye, and the control unit includes an ocular dynamics monitoring processing unit that monitors the movement of the subject's eye based on the pupil circle and the bright spot. [8] [7] The ophthalmic device according to the present invention, The ophthalmologic apparatus is characterized in that the ocular dynamics monitoring processing unit includes a second extraction unit that uses output information from the event camera to extract, as the pupil circle, a contour area whose brightness changes due to a difference in brightness with the cornea (black part of the eye) when the position of the pupil of the subject's eye changes, and extracts, as the bright point, a point area whose brightness changes when the direction of the gaze of the subject's eye changes, and a second determination unit that determines the movement of the subject's eye based on the extraction results of the pupil circle and the bright point by the second extraction unit. [9] [8] The ophthalmic device according to the present invention, The second determination unit monitors the shape of the pupil circle and the bright spot extracted by the second extraction unit, and when the pupil circle and the bright spot disappear or are missing, determines that the subject's eye is blinking or has ptosis.

[10] In the ophthalmic device described in [8] or [9], The ophthalmologic apparatus is characterized in that the second determination unit monitors changes in the positions of the pupil circle and the bright spot extracted by the second extraction unit, and determines the line of sight of the subject's eye based on the changes in the positions of the pupil circle and the bright spot. [Explanation of symbols]

[0108] A Ophthalmology equipment 10 Stand 20 Main body 30 Chin rest (face support) 33 Forehead support part (face support part) 50 Optical system 60 Control Unit 64 Facial movement monitoring processing unit 641 1st extraction part 642 1st Judgment Department 65 Eye movement monitoring processing unit 651 2nd extraction part 652 Second Judgment Section 80 Event Camera 80a Right Event Camera 80b Left event camera E. Examined eye J. Jaw IC eye corner OC outer corner of eye EP pupillary circle BS bright spot

Claims

1. an optical system for acquiring ocular information of a subject's eye; a face support portion that supports the face of the subject; a control unit that controls each unit of the device; In an ophthalmic apparatus comprising: an event camera is attached at a position capturing the anterior segment of the subject's eye, and the event camera outputs only event data in which a luminance change exceeds a set threshold value among pixel data of an anterior segment image frame of the subject's eye, in combination with coordinate and time information; The control unit includes a facial movement monitoring processing unit that extracts an inner corner point and an outer corner point of the eye of the subject using output information from the event camera and monitors facial movements of the subject based on the inner corner point and the outer corner point of the eye. An ophthalmic device characterized by:

2. 2. The ophthalmic apparatus according to claim 1, the face movement monitoring processing unit is a first extraction unit that uses output information from the event camera to extract upper eyelid lines and lower eyelid lines whose luminance changes along line segments of the upper eyelid and lower eyelid due to a difference in luminance between the eyelid and the sclera (white of the eye) of the subject's eye when the subject's face moves, and extracts the inner corner point and the outer corner point of the eye of the subject's eye from two intersections of the upper eyelid line and the lower eyelid line; a first determination unit that determines a movement of the face of the subject based on the extraction result of the inner corner point and the outer corner point of the eye by the first extraction unit. An ophthalmic device characterized by:

3. 3. The ophthalmic apparatus according to claim 2, The first determination unit monitors the movement of the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and determines that the face of the subject is separated from the face support unit when the inner corner point and the outer corner point of the eye are moving in the same direction on a two-dimensional coordinate plane defined by a left-right axis and a top-bottom axis. An ophthalmic device characterized by:

4. 3. The ophthalmic apparatus according to claim 2, The first determination unit monitors a change in the distance between the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and determines that an increase in the distance indicates a movement of the subject's face approaching in the anterior-posterior axis direction, and that a decrease in the distance indicates a movement of the subject's face receding in the anterior-posterior axis direction. An ophthalmic device characterized by:

5. 3. The ophthalmic apparatus according to claim 2, The first determination unit monitors the movement of the inner corner point and the outer corner point of the eye extracted by the first extraction unit relative to the center of the eye, and determines that the face of the subject is tilted in the direction of the rotation when the inner corner point and the outer corner point of the eye are rotating relative to the center. An ophthalmic device characterized by:

6. 3. The ophthalmic apparatus according to claim 2, the event cameras are a right event camera and a left event camera that are stereo-arranged so as to capture an anterior segment of one of the examinee's eyes from two different directions, The first determination unit calculates three-dimensional position coordinates for each of the inner corner point and the outer corner point of the eye extracted by the first extraction unit, and determines that a direction perpendicular to a line connecting the inner corner point and the outer corner point of the eye in a three-dimensional coordinate space is a face direction. An ophthalmic device characterized by:

7. 7. The ophthalmic apparatus according to claim 1, The control unit includes an ocular dynamics monitoring processing unit that extracts a pupil circle representing the pupil shape of the subject's eye and a bright spot projected onto the cornea of ​​the subject's eye from the optical system using output information from the event camera, and monitors the movement of the subject's eye based on the pupil circle and the bright spot. An ophthalmic device characterized by:

8. 8. The ophthalmic apparatus according to claim 7, the eye movement monitoring processing unit uses output information from the event camera to extract, as the pupil circle, a contour region whose luminance changes due to a difference in luminance with the cornea (iris region) when the position of the pupil of the subject's eye changes, and extracts, as the bright point, a point region whose luminance changes when the direction of the line of sight of the subject's eye changes; a second determination unit that determines a movement of the subject's eye based on the extraction results of the pupil circle and the bright spot by the second extraction unit. An ophthalmic device characterized by:

9. 9. The ophthalmic apparatus according to claim 8, The second determination unit monitors the shapes of the pupil circle and the bright spot extracted by the second extraction unit, and determines that the subject's eye is blinking or has ptosis when the pupil circle and the bright spot disappear or are missing. An ophthalmic device characterized by:

10. 9. The ophthalmic apparatus according to claim 8, The second determination unit monitors changes in the positions of the pupil circle and the bright spot extracted by the second extraction unit, and determines the line of sight of the subject's eye based on the changes in the positions of the pupil circle and the bright spot. An ophthalmic device characterized by:

Images