Ophthalmic device

Through learning the AI ​​image recognition technology of the model, the problem of difficulty in pupil detection in existing ophthalmic devices is solved, and the robust detection of the pupil area and the stability of automatic alignment is achieved.

CN119055186APending Publication Date: 2024-12-03TOPCON CORPORATION
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410615492.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-05-17
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

During pupil detection, the pupil is blocked due to the contact of eyelids or eyelashes, making it difficult to detect the pupil area correctly and stably. Especially in wide-angle shooting, the pupil shape becomes slender, further increasing the detection difficulty.

Method used

Using AI image recognition technology for learning completion model, the pupil area prediction model is generated by pre-collecting teacher data of pupil area information, input anterior eye camera image data for inference execution, detect pupil area, and eliminate the impact of obstacles such as mapping.

Benefits of technology

The robustness in pupil detection is achieved, the pupil area can be identified correctly and stably, and the stability of automatic alignment is improved, especially in the presence of mapping or pupil shape changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119055186A_ABST
    Figure CN119055186A_ABST
Patent Text Reader

Abstract

The present invention provides an ophthalmic apparatus capable of stably detecting a pupil region by using AI image recognition using a learning completion model at the time of pupil detection. An ophthalmic device (A) is provided with an anterior eye stereo camera (22) that captures an image of an anterior eye of a subject eye (E), and is provided with a learning completion model setting unit (621) and a pupil detection processing unit (632). A learning completion model setting unit (621) prepares a plurality of pieces of teacher data in which pupil region information is added to anterior eye camera image data, and sets a learning completion pupil region prediction model generated by a learning process in which the teacher data is read into a selected machine learning model. A pupil detection processing unit (632) detects the pupil region of the subject eye (E) on the basis of pupil region prediction information that is model-output by performing reasoning in which anterior eye camera image data acquired by an anterior eye stereo camera (22) is input to a learned pupil region prediction model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an ophthalmic device. Background Art

[0002] In the past, an ophthalmic device is known for the purpose of providing an ophthalmic device that can properly align the examined eye with the optical system of the device, wherein a stereo camera is arranged on both sides of an objective lens as an anterior ocular camera, and the anterior ocular portion of the examined eye is simultaneously photographed from different directions (see patent document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-248376 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, when the eye characteristics of the examined eye are examined or photographed with the subject supporting the lower jaw on the jaw rest, the position of the examined eye will shift every time the subject changes, and therefore it is necessary to adjust the alignment control (positioning control) of the relative position relationship between the examined eye and the main body (= device optical system).

[0008] However, the stereo camera images captured by the stereo camera are prone to vignetting, in which the pupil is blocked by the reflection of the eyelids or eyelashes. In particular, in the case of an ophthalmic device such as a fundus camera with a wide-angle shooting angle, due to the optical constraints accompanying the widening of the shooting angle, the lens diameter of the objective lens increases, and the camera installation angle becomes an acute angle compared to the camera installation angle in the case of a small lens diameter. Therefore, the pupil shape of the eye to be examined becomes an elongated elliptical shape, which makes it easier to produce vignetting, in which the pupil is blocked by the reflection of the eyelids or eyelashes.

[0009] In this regard, the existing pupil area detection method adopts the following method, that is, searching for a bright spot from a stereo camera image, and if no bright spot is found, binarizing the entire image using the brightness threshold of the pupil with the lowest brightness in the identification image, and if a bright spot is found, binarizing the image around the bright spot using the brightness threshold of the pupil with the lowest brightness in the identification image, and detecting from the binarized image. Therefore, in the case of the existing technology of detecting the pupil area using a stereo camera image, if there is a reflection of eyelids or eyelashes, etc., the image becomes a pupil area shape in which part of the pupil area is missing due to binarization. In this way, there is the following technical problem, that is, if there is a portion in the pupil image with a brightness level higher than the pupil brightness, the portion with higher brightness is not recognized as the pupil area, and the roundness or area of ​​the recognized pupil area is reduced, making it difficult to detect the pupil area correctly and stably.

[0010] The present disclosure is proposed with a view to the above-mentioned technical problems, and its object is to provide an ophthalmic device that can robustly detect the pupil area by using AI image recognition of a learning completion model during pupil detection.

[0011] Means used to solve problems

[0012] In order to achieve the above-mentioned purpose, the ophthalmic device disclosed in the present invention has an anterior ocular camera for photographing the anterior segment of an eye to be examined. It has: a learning model setting unit for setting a learning model related to the anterior ocular image of the eye to be examined; and a pupil detection processing unit for detecting the pupil area of ​​the eye to be examined. The learning model setting unit prepares a plurality of teacher data with pupil area information attached to pre-collected anterior ocular camera image data, and sets a learning pupil area prediction model generated by making a selected machine learning model read the learning process of the teacher data. The pupil detection processing unit detects the pupil area of ​​the eye to be examined based on the prediction information of the pupil area output by the model (output from the learning pupil area prediction model) by inputting the anterior ocular camera image data acquired from the anterior ocular camera into the inference execution of the learning pupil area prediction model.

[0013] Effects of the Invention

[0014] In the ophthalmic device of the present disclosure, the pupil area detection information is not the anterior ocular camera image information which is affected by reflections and the like which hinder the pupil detection, but is the pupil area prediction information output by the model through inference execution from the learned pupil area prediction model. Therefore, when detecting the pupil, the pupil area can be robustly detected by AI (Artificial Intelligence) image recognition using the learned model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a perspective view showing the external appearance structure of the ophthalmologic device of Example 1 as viewed from the jaw rest side.

[0016] Figure 2 This is a perspective view showing the external appearance structure of the ophthalmologic apparatus according to Example 1 as viewed from the control panel side.

[0017] Figure 3 This is a front view showing the external appearance structure of the front side of the main body of the ophthalmologic apparatus according to Example 1, as viewed from the jaw rest side.

[0018] Figure 4 This is a side view showing the schematic structure of the internal components and accessories of the ophthalmologic apparatus according to the first embodiment.

[0019] Figure 5This is a schematic diagram showing the configuration of an optical system in the ophthalmologic apparatus according to Example 1.

[0020] Figure 6 This is a block diagram showing the configuration of a control system in the ophthalmologic apparatus according to the first embodiment.

[0021] Figure 7 This is a flowchart showing the flow of basic operations when the ophthalmologic apparatus of the first embodiment captures any one of an anterior segment image, a fundus image, and a fundus tomographic image of the eye to be examined.

[0022] Figure 8 This is a schematic diagram showing the structure of PSPNet selected as a machine learning model in the pupil detection process of the pupil detection processing unit of Example 1.

[0023] Fig. 9 This is a flowchart showing the flow of pupil detection processing in the pupil detection processing unit of the first embodiment.

[0024] Fig.10 This is a flowchart showing the flow of pupil detection processing for detecting the pupil of the eye to be examined prior to automatic alignment of the pupil in the background art.

[0025] Fig.11A is an image explanation diagram showing an example of anterior ocular camera image in pupil detection processing according to the background art. Fig. 11B is an image explanation diagram showing an example of searching for a bright spot from an anterior ocular image, Fig. 11C is an image explanation diagram showing an example of binarization of the periphery of a bright spot. Fig.11D is an image explanation diagram showing an example of acquiring pupil center coordinates by applying ellipse approximation to the contour edge.

[0026] Fig. 12A is an image explanation diagram showing an example of an anterior ocular camera image in pupil detection processing according to Embodiment 1, Fig. 12B is an image explanation diagram showing an example of a pupil probability map, Fig. 12C is an image illustration showing an example of a pupil region with a pupil label in a pupil candidate region, Fig.12D is an image explanation diagram showing an example of acquiring pupil center coordinates by applying ellipse approximation.

[0027] Fig.13 This is a perspective view showing an objective lens unit equipped with an anterior ocular segment three-direction camera composed of three cameras in the ophthalmologic apparatus of Example 2.

[0028] Description of Reference Numerals

[0029] A: Ophthalmic devices,

[0030] 10: Bracket part,

[0031] 18: XYZ drive unit,

[0032] 20: Main body,

[0033] 22: Anterior ocular stereo camera (anterior ocular camera),

[0034] 22a: right camera,

[0035] 22b: Left camera,

[0036] 50: Optical system,

[0037] 51: objective lens,

[0038] 60: Control Department,

[0039] 62: Storage Department,

[0040] 621: Learning completed model setting department,

[0041] 63: Alignment control unit,

[0042] 632: pupil detection processing unit,

[0043] 633: Automatic alignment unit,

[0044] E: Eye under examination,

[0045] Ea: anterior eye,

[0046] Ef: fundus. DETAILED DESCRIPTION

[0047] Based on Example 1 and Example 2 shown in the accompanying drawings, a method for implementing the ophthalmic device disclosed in the present invention is described. Example 1 and Example 2 are application examples of an ophthalmic device that observes, photographs, and records an image of the anterior segment of an eye under examination, an image of the fundus of an eye under examination, and a tomographic image of the fundus of an eye under examination, and provides them as electronic images for diagnosis. In addition, in each of the accompanying drawings, with the eye under examination as a reference, the left-right axis in the left-right direction (horizontal direction) relative to the main body of the ophthalmic device is indicated by the X-axis, the up-down axis in the up-down direction (vertical direction) is indicated by the Y-axis, and the front-back axis in the front-back direction (depth direction) orthogonal to the X-axis and the Y-axis is indicated by the Z-axis.

[0048] [Example 1]

[0049] [Overall structure of the device ( Figure 1 to Figure 4 )]

[0050] like Figure 1 to Figure 4 As shown, the ophthalmologic apparatus A includes a stand portion 10 , a main body portion 20 , a jaw rest portion 30 , a control panel portion 40 , an optical system 50 , and a control portion 60 .

[0051] The ophthalmic device A includes a fundus camera for acquiring fundus images of the inspected eye E and an OCT (abbreviation of "Optical Coherence Tomography") for acquiring fundus tomographic images of the inspected eye E. Here, the "fundus camera" refers to a camera that images the fundus conditions such as the retina, optic nerve, capillaries, etc. located deep in the inspected eye E and captures fundus images. "OCT" refers to an optical coherence tomography that uses interference of light to capture tomographic images of the retina present in the fundus of the inspected eye E and captures fundus tomographic images.

[0052] like Figure 4 As shown, the bracket portion 10 is placed on a height-adjustable optometry table T, etc. The main body portion 20 is supported on the upper surface of the bracket portion 10 so as to be movably in the three-axis directions of the X-axis, the Y-axis, and the Z-axis. A jaw rest portion 30 is fixed to the front surface of the bracket portion 10. A power switch 11, a power socket 12, a USB terminal 13, and a LAN terminal 14 are provided on the side of the bracket portion 10. In addition, USB is the abbreviation of "Universal Serial Bus", and LAN is the abbreviation of "Local Area Network". The USB terminal 13 is a terminal for connecting an external memory, and is connected to a HDD (abbreviation of "Hard Disk Drive") or a USB memory, etc. The LAN terminal 14 is connected to a personal computer 16 installed with dedicated software, etc. via a LAN cable 15.

[0053] like Figure 4 As shown, a power supply unit 17 and an XYZ drive unit 18 are built into the internal space of the bracket unit 10. The power supply unit 17 includes a power switch 11, a power socket 12, a USB terminal 13, and a LAN terminal 14. The XYZ drive unit 18 is a motor actuator having a motor and a motor drive circuit that drives the main body 20 in the three-axis direction (three-dimensional direction) of the XYZ axis when the main body 20 moves relative to the bracket unit 10 in alignment control.

[0054] The main body 20 is configured to be movable in the X-axis direction, the Y-axis direction, and the Z-axis direction relative to the bracket 10 to which the jaw rest 30 is fixed, by means of the XYZ driving unit 18. In the main body 20, an optical system 50 and a control unit 60 for performing an ocular characteristic test of the subject's eye E or observing the anterior ocular segment or fundus of the subject's eye E while the subject's lower jaw is supported on the jaw rest 30 are built into a main body cover 21 covering the entire body. Figure 1 , 2 As shown in FIG. 4 , a control panel portion 40 is disposed at an upper position on the back side of the main body cover 21 .

[0055] like Figure 3As shown, at the front surface position of the main body cover 21, there is an objective lens 51 of the optical system 50 in the center portion facing the subject's eye E. In addition, there are an anterior ocular stereo camera 22 (an example of an anterior ocular camera), a peripheral fixation lamp 23, and an anterior ocular observation filter 24 at the periphery of the objective lens 51.

[0056] The anterior ocular stereo camera 22 is a camera that obtains an anterior ocular image by photographing the anterior ocular of the subject. The anterior ocular stereo camera 22 is composed of two right cameras 22a and left cameras 22b arranged at both sides of the objective lens 51 so that the lens optical axis is tilted toward the anterior ocular of the subject's eye E, which is the object of inspection or observation. The right camera 22a and the left camera 22b obtain the right anterior ocular image and the left anterior ocular image of the subject's face supported by the chin rest 30 cut off according to the selection of the subject's eye E and the viewing angle at this time. In addition, the anterior ocular stereo camera 22 is composed of two right cameras 22a and left cameras 22b configured to determine the installation width size and installation angle in the X-axis direction. Therefore, if two two-dimensional pupil position information is obtained based on the two anterior ocular images, the three-dimensional coordinate position of the pupil can be determined by using known information or calculation processing of well-known trigonometric functions.

[0057] The peripheral fixation lamps 23 are fixation lamps for fixing the sight line of the inspected eye E by lighting, and eight of them are arranged at equal intervals at the peripheral position of the objective lens 51. The anterior segment observation filter 24 is a filter used for adjusting the light amount during anterior segment observation or anterior segment OCT, and two of them are arranged in the longitudinal direction of the outer position of the right camera 22a and the outer position of the left camera 22b (a total of four).

[0058] The jaw rest 30 is provided so that its height position (position in the Y-axis direction) can be adjusted relative to the jaw rest support part 31 of the fixed frame part 10, and supports the lower jaw of the subject. The jaw rest 30 includes a lifting rod 30a that is lifted and lowered by a built-in jaw rest driving part 32, a jaw rest base 30b fixed at the upper end position of the lifting rod 30a, and jaw rest stopper pins 30c provided at both sides of the jaw rest base 30b. The jaw rest driving part 32 is a motor actuator having a motor and a motor driving circuit that drives the lifting rod 30a in the Y-axis direction when the jaw rest 30 is moved relative to the jaw rest support part 31 (=frame part 10) in the Y-axis direction during alignment control.

[0059] The chin rest support portion 31 has a face support frame portion 33 fixed at both end positions of the T-shape, and the face support frame portion 33 is shaped so as to surround the face of the subject supporting the lower jaw on the chin rest portion 30 in three directions. A height mark 33a serving as a reference for the height position of the eye E to be examined is provided on a pair of vertical frames extending in the Y-axis direction in the face support frame portion 33. A detachable forehead stopper surface 33b formed of silicone rubber or the like is provided on a grade frame connecting the upper ends of a pair of vertical frames in the face support frame portion 33. Furthermore, an arm 34 that can be bent in multiple stages is provided at the central upper position of the grade frame in the face support frame portion 33, and an external fixation mark 35 is provided at the front end of the arm 34.

[0060] The control panel unit 40 is arranged at the upper back side of the main body cover 21, and has a display screen 41 that displays in color the anterior ocular image of the subject's eye E from the anterior ocular stereo camera 22 or the anterior ocular observation image of the subject's eye E from the optical system 50. The display screen 41 is a touch panel on which the examinee can perform touch operations on the displayed button images and images with a finger as input operations to the control unit 60. The control panel unit 40 is a combined support structure of a bending support and a rotation support in connection with the support unit 42 of the main body 20. Through this support structure, the display screen 41 can be set at any position in the entire circumferential direction relative to the main body 20, and the tilt angle of the display screen 41 can also be freely set.

[0061] When the examiner performs an eye characteristic test by remote operation from a position away from the examinee, a remote operation tablet 40' is used instead of the control panel unit 40. The remote operation tablet 40' has a display screen 41' based on a touch panel, and in addition to having the same input operation function as the control panel unit 40, it also has a communication function with the main body 20.

[0062] The optical system 50 is a system for observing the subject's eye E or inspecting the eye characteristics of the subject's eye E in a state where the subject supports the lower jaw on the jaw rest 30. Figure 4 As shown, the optical system 50 includes a fundus camera unit 52 having an objective lens 51 and an OCT unit 53. The fundus camera unit 52 includes an illumination optical system and a photographing optical system, and is a unit constituting a fundus camera that acquires a fundus image of the subject's eye E by means of a lens and an imaging element. The OCT unit 53 is a unit constituting an OCT that acquires a fundus tomographic image of the subject's eye E by means of a wavelength-variable light source or an optical fiber coupler. In addition, in addition to being able to acquire a fundus image and a fundus tomographic image of the subject's eye E, an anterior ocular observation image of the subject's eye E can also be acquired from the optical system 50. The detailed structure of the optical system 50 will be described later.

[0063] The control unit 60 controls various parts of the device (the fundus camera unit 52, the OCT unit 53, the jaw rest unit 30, the main body unit 20, etc.) based on various input operations including touch operations on the display screen 41 of the control panel unit 40. Figure 4 As shown, the control unit 60 includes a control substrate 60a, a CPU (Central Processing Unit) substrate 60b, and an image board 60c as a hardware configuration.

[0064] [Optical system structure ( Figure 5 )]

[0065] like Figure 5 As shown, the optical system 50 includes a fundus camera unit 52 having an objective lens 51 and an OCT unit 53. The fundus camera unit 52 has an alignment optical system 521, a photographing optical system 522, a focusing optical system 523, an illumination optical system 524, and an OCT optical system 525.

[0066] The alignment optical system 521 generates an alignment bright spot by projecting the output light from the light source, i.e., LED 521a, onto the anterior segment Ea (= corneal surface) and the fundus Ef of the subject's eye E via the objective lens 51. That is, the output light (alignment light) from the LED 521a is reflected by the dichroic mirror 521e via the apertures 521b, 521c and the relay lens 521d. Then, the output light passes through the hole of the aperture reflector 524k, passes through the dichroic mirror 525g, and is projected onto the anterior segment Ea and the fundus Ef of the subject's eye E via the objective lens 51.

[0067] The photographing optical system 522 acquires the front image of the anterior segment and the front image of the fundus of the subject's eye E by means of a CCD (Charge Coupled Device) image sensor 522a. The imaging lens 522b focuses on the anterior segment Ea when the optical path is inserted, and focuses on the fundus Ef when the optical path of the photographing optical system 522 is separated. That is, the anterior segment reflected light of the aiming light passes through the objective lens 51 and the dichroic mirror 525g, passes through the hole of the aperture reflector 524k, and a part of it passes through the dichroic mirror 521e. Then, it passes through the focusing lens 522e, the condensing lens 522c, and the imaging lens 522b, and is projected onto the CCD image sensor 522a. On the other hand, the fundus reflected light of the aiming light passes through the objective lens 51 and the dichroic mirror 525g, passes through the hole of the aperture reflector 524k, and a part of it passes through the dichroic mirror 521e. Then, it passes through the focus lens 522e and the condenser lens 522c and is projected onto the CCD image sensor 522a.

[0068] The focusing optical system 523 generates a mark (spectrometric mark) for focusing (focus) relative to the fundus Ef. When focusing, the reflection surface of the reflection component 523g is arranged obliquely on the optical path of the illumination optical system 524. The light (focused light) output from the LED 523a of the focusing optical system 523 passes through the relay lens 523b and is separated into two light beams by the spectroscopic mark plate 523c. Then, it passes through the two-hole aperture 523d, is reflected by the reflection mirror 523e, and is temporarily imaged and reflected on the reflection surface of the reflection component 523g by the focusing lens 523f. Further, the focused light is reflected by the aperture reflection mirror 524k via the relay lens 524j, passes through the dichroic mirror 525g, is refracted by the objective lens 51, and is projected onto the fundus Ef. In addition, the fundus reflected light of the focused light is detected by the CCD image sensor 522a through the same path as the fundus reflected light of the alignment light.

[0069] The illumination optical system 524 irradiates the fundus Ef with observation illumination light. The light (observation illumination light) output from the observation light source 524a is reflected by the reflector 524b having a curved reflection surface, passes through the condenser lens 524c, and passes through the visible cut filter 524d to become near infrared light. Furthermore, the observation illumination light is temporarily converged near the shooting light source 524E, reflected by the reflector 524f, and passes through the relay lenses 524g, 524h, the aperture 524i, and the relay lens 524j. Then, the observation illumination light is reflected at the peripheral portion (surrounding area of ​​the hole portion) of the aperture reflector 524k, passes through the dichroic mirror 525g, and is refracted by the objective lens 51 to illuminate the fundus Ef. The fundus reflected light of the observation illumination light is refracted by the objective lens 51, passes through the dichroic mirror 525g, passes through the hole portion formed in the central area of ​​the aperture reflector 524k, and passes through the dichroic mirror 521e. Then, it is projected onto the CCD image sensor 522a via the focus lens 522e and the condenser lens 522c. In addition, when the photographing optical system 522 is focused on the anterior ocular segment Ea by inserting the imaging lens 522b, an observed image of the anterior ocular segment of the subject's eye E is displayed.

[0070] The OCT optical system 525 forms an optical path for OCT measurement that guides the signal light from the OCT unit 53 to the fundus Ef via the photographing optical system 522 and guides the reflected light from the fundus Ef to the OCT unit 100 via the photographing optical system 522. The optical path for OCT measurement includes a collimating lens unit 525a, an optical path length changing unit 525b, a galvanometer scanner 525c, a focus lens 525d, a reflecting mirror 525e, a relay lens 525f, and a dichroic mirror 525g. The dichroic mirror 525g branches the optical path for OCT measurement from the optical path for fundus imaging, reflects light of a wavelength band used for OCT measurement, and transmits light for anterior segment imaging and fundus imaging.

[0071] [Control System Structure( Figure 6 )]

[0072] like Figure 6 As shown, the control system of the ophthalmologic apparatus A includes a control unit 60 . The XYZ drive unit 18 , the anterior ocular stereo camera 22 , the chin rest drive unit 32 , the control panel unit 40 , and the optical system 50 are connected to the control unit 60 .

[0073] The control unit 60 includes a main control unit 61 for controlling the fundus camera unit 52 and the OCT unit 53, a storage unit 62 for storing necessary data, and an alignment control unit 63. The alignment control unit 63 performs alignment control for adjusting the relative positional relationship between the eye E to be examined and the main unit 20 (objective lens 51) based on the anterior ocular segment image acquired by the anterior ocular segment stereo camera 22 and the anterior ocular segment front image and the fundus front image acquired by the optical system 50. The storage unit 62 includes a learning completion model setting unit 621. The alignment control unit 63 includes a chin rest height adjustment unit 631, a pupil detection processing unit 632, an automatic alignment unit 633, and a manual alignment unit 634.

[0074] The alignment control unit 63 obtains an anterior ocular image by photographing the left anterior ocular segment and the right anterior ocular segment of the face of the subject from two different directions by the anterior ocular stereo camera 22 (right camera 22a, left camera 22b). In addition, the anterior ocular segment front image and the fundus front image are obtained by photographing the right eye and the left eye of the subject's eye E respectively by the optical system 50. The alignment control unit 63 adjusts the relative position relationship between the subject's eye E and the objective lens 51 provided in the main body 20 by outputting a drive instruction to at least one of the XYZ drive unit 18 and the jaw support drive unit 32. Here, the XYZ drive unit 18 and the jaw support drive unit 32 are used separately. If the adjustment movement amount is only the movement amount in the XZ axis direction, the XYZ drive unit 18 is used. On the other hand, when the adjustment movement amount includes the movement amount in the Y-axis direction, since the Y-axis movement allowable range of the jaw rest driving unit 32 is larger than the movement allowable range of the XYZ driving unit 18, the XYZ driving unit 18 and the jaw rest driving unit 32 are used separately. For example, in the case of Y-axis movement, the jaw rest driving unit 32 is used for jaw rest height adjustment, and the XYZ driving unit 18 is used for automatic alignment.

[0075] When confirming that the examined eye E is reflected in the anterior segment image displayed on the screen, the chin rest height adjuster 631 adjusts the height of the chin rest 30 via the chin rest driving unit 32 so that the pupil center of the examined eye E is located near the center of the anterior segment image.

[0076] The height adjustment of the chin rest is performed by the examiner manually operating the display screen 41 while observing the anterior ocular front image and the anterior ocular image displayed on the control panel 40. The specific operation is as follows, that is, in the anterior ocular front image displayed on the display screen 41, the examiner clicks on the displayed pupil to make the pupil enter the display frame of the anterior ocular front image. Then, in the anterior ocular image obtained by the camera close to the photographing eye displayed on the display screen 41, the examiner touches the displayed chin rest up and down movement button to align the height mark 33a of the chin rest 30 with the height of the examined eye E based on the pupil display line.

[0077] The pupil detection processing unit 632 detects the pupil area of ​​the inspected eye E based on the prediction information of the pupil area output by the model through the inference execution. In the inference execution, the anterior ocular camera image data acquired from the anterior ocular stereo camera 22 is input into the learned pupil area prediction model. Here, the "learned pupil area prediction model" refers to a learned model generated by preparing a plurality of teacher data in which pupil area information is added to the pre-collected anterior ocular camera image data, and making the selected machine learning model (AI model) read the learning process of the teacher data. The learned pupil area prediction model is read from the learned model setting unit 621 of the storage unit 62. Here, the reason why the learned model setting unit 621 is set in the storage unit 62 of the ophthalmic device A is that edge computing (one of the network technologies of configuring edge processing devices near the device) is required in order to process the automatic alignment of the pupil in real time. The anterior ocular camera image data are acquired from the image sensor (not shown) of the right camera 22a and the image sensor (not shown) of the left camera 22b of the anterior ocular stereo camera 22, respectively.

[0078] Here, the generation of a pupil area prediction model after learning is described. First, a machine learning model suitable for pupil area prediction is selected. In Example 1, a pyramid parsing network model, namely PSPNet (abbreviation for "Pyramid Scene Parsing Network"), is selected as a machine learning model. Then, with the selection of PSPNet, a method of marking feature areas for each pixel (Pixel), namely semantic segmentation, is adopted. For example, a manual annotation method is used to generate teacher data with pupil area information attached to the anterior eye camera image data. That is, prepare a plurality of teacher data with pupil area information attached in pixel units to the pre-collected anterior eye camera image data. The teacher data is data of the original pupil area where the pupil is located attached to the camera image data, and even if eyelids or eyelashes are reflected in the collected anterior eye camera image data, this is ignored. Regarding the number of teacher data, prepare the number of data required to obtain the prediction accuracy of the target pupil area (for example, hundreds of data). Then, a learning-completed pupil area prediction model is generated through a learning process in which the selected PSPNet reads a plurality of teacher data whose data sizes are unified through normalization.

[0079] Regarding the number of generated learned pupil area prediction models, the goal is not to generate a learned universal model for different ophthalmic devices, but to generate a learned universal model for various ophthalmic devices. For example, regardless of the type or type of ophthalmic device, multiple anterior ocular stereo camera image data are collected, pupil area information is added to the anterior ocular stereo camera image data to generate teacher data, and the teacher data is used to generate a learned pupil area prediction model that is common to various ophthalmic devices. However, in order to cope with the different configurations of the anterior ocular cameras due to different device designs, the anterior ocular camera image data collected separately according to the different camera configurations can also be used to generate learned pupil area prediction models for different devices or different camera configurations. In addition, the detailed pupil detection processing in the pupil detection processing unit 632 is described later.

[0080] When the anterior ocular stereo camera 22 is used, if pupil detection based on camera image data from two directions is successful in the pupil detection processing unit 632 , the automatic alignment unit 633 performs automatic alignment of the pupils based on the two successful pupil center coordinates.

[0081] Here, the execution of automatic alignment of the pupil is described. First, the three-dimensional current coordinates (xo, yo, zo) of the center of the pupil are calculated. Next, the three-dimensional target coordinates (xt, yt, zt) of the center of the pupil and the calculated three-dimensional current coordinates (xo, yo, zo) are used to calculate the X-axis difference Δx (= xt-xo), the Y-axis difference Δy (= yt-yo), and the Z-axis difference Δz (= zt-zo), respectively. Next, a drive instruction for moving the main body 20 according to the X-axis difference Δx, the Y-axis difference Δy, and the Z-axis difference Δz is output to the XYZ drive unit 18. Next, it is determined whether the X-axis difference Δx, the Y-axis difference Δy, and the Z-axis difference Δz after the main body 20 is moved by the output of the drive instruction are respectively less than the alignment threshold (for example, a value of about ±0.2 mm). During the period when it is determined that the X-axis difference Δx, the Y-axis difference Δy, and the Z-axis difference Δz exceed the alignment threshold, the difference calculation and driving are repeated. Then, if it is determined that the X-axis difference Δx, the Y-axis difference Δy, and the Z-axis difference Δz are each less than the alignment threshold, the automatic alignment of the pupil is terminated.

[0082] The calculation of the three-dimensional current coordinates (xo, yo, zo) of the pupil center is described. First, the two pupil center coordinates of the successful pupil detection are obtained from the pupil detection processing unit 632. Then, using the two obtained pupil center coordinates and the known left and right camera installation spans, the known camera installation positions, and the known setting angles of the camera lens optical axis, a triangle connecting the three points of the current pupil center position, the left camera position, and the right camera position is drawn in three-dimensional space. Then, the three-dimensional current coordinates (xo, yo, zo) of the pupil center are calculated using known trigonometric functions on the drawn triangle. In addition, among the three-dimensional target coordinates (xt, yt, zt) of the pupil center, the position of the anterior eye focused on the subject's eye E in the Z-axis direction is set as the target coordinate zt of the pupil center on the Z axis. Then, in the XY coordinate plane at the position of the target coordinate zt, the position where the optical axis OA of the objective lens 51 coincides with the pupil center is set as the target coordinates xt and yt of the XY axis. In this way, automatic alignment of the pupil is performed on the condition that pupil detection from two directions is successful for the photographing eye.

[0083] When the automatic alignment is not completed even if the elapsed time from the start reaches the time limit, or when the examiner actively selects manual operation, the manual alignment unit 634 performs manual alignment control through the examiner's manual operation on the display screen 41 .

[0084] In manual alignment control, if the manual mode button displayed on the automatic alignment screen is clicked, the automatic alignment of the pupil is terminated and the mode is shifted to the "manual adjustment mode" for manually adjusting the photographing eye. In the "manual adjustment mode", the pupil marks of the two anterior ocular images displayed on the display screen 41 of the control panel unit 40 are clicked. By driving the XYZ drive unit 18 based on the click operation, the XYZ axis alignment adjustment is performed so that the two pupil marks are arranged at the center of the anterior ocular image in an overlapping manner.

[0085] [Photographing and processing operation of the image of the inspected eye ( Figure 7 )]

[0086] One side reference Figure 7 The flowchart shown in FIG. 1 is used to explain the photographing processing operation when photographing an image of the inspected eye (for example, an anterior ocular segment image, a fundus image, or a fundus tomographic image) executed by the control unit 60. The photographing processing operation is started after the inspected person confirms that the ophthalmologic apparatus A has turned on the power switch.

[0087] In step S1, after the start, patient registration based on the name or patient ID of the patient (= subject) is performed. Here, "patient ID" refers to an identification number for managing the personal information of the subject's ophthalmic examination, which can include age, gender, past examination information for tracking, etc.

[0088] In step S2 , following the patient registration in step S1 , the chin rest height is adjusted by a manual operation of the examinee. The chin rest height adjustment is performed by the chin rest height adjustment unit 631 of the alignment control unit 63 .

[0089] In step S3, following the chin rest height adjustment in step S2, the type of photographing is selected. The type of photographing is selected by selecting any one of the photographing modes (anterior ocular segment image photographing mode, fundus image photographing mode, fundus tomographic image photographing mode, etc.) from the photographing icon selection screen displayed on the display screen 41 of the control panel unit 40 by touch operation. In addition, regarding the selection of the photographing eye, the photographing eye is selected by touching the display button together with the selection of the photographing type.

[0090] In step S4 , following the selection of the imaging type in step S3 , pupil detection processing is performed on the subject's eye E (=imaging eye). The pupil detection processing is performed by the pupil detection processing section 632 of the alignment control section 63 .

[0091] In step S5, after the pupil detection from two directions in the pupil detection process in step S4 is successful, automatic alignment (automatic adjustment) of the pupil is performed. The automatic alignment is performed by the automatic alignment unit 633 of the alignment control unit 63. In addition, when the automatic alignment time becomes long or the detector actively wishes, the manual alignment (manual adjustment) of the pupil is performed by the manual alignment unit 634 instead of the automatic alignment of the pupil.

[0092] In step S6, following the automatic alignment in step S5 or the preview confirmation NG in step S8, automatic focusing of automatic focus adjustment is performed. In this automatic focusing, the focus is adjusted on the anterior segment Ea of the subject's eye E in the anterior segment image capturing mode, and the focus is adjusted on the fundus Ef of the subject's eye E in the fundus image capturing mode and the fundus tomographic image capturing mode.

[0093] In step S7, following the automatic focusing in step S6, an image of the eye to be examined (for example, an image of the anterior ocular segment, an image of the fundus, or an image of the fundus tomography) is captured. In this capturing, if the "OK button" is clicked on the capturing screen displayed on the display screen 41, the current capturing is performed and the next capturing is transferred. In addition, a preview of the captured image is displayed at each capturing.

[0094] In step S8, after shooting in step S7, the preview of the shot image is confirmed to determine whether the preview confirmation is OK (qualified) or preview confirmation NG (unqualified). If it is determined that the preview confirmation is OK, the process proceeds to step S9, and if it is determined that the preview confirmation is NG, the process returns to step S6. That is, if it is determined that the preview confirmation is NG, the process returns to step S6 to perform autofocus again, and until the preview confirmation is determined to be OK in step S8, shooting can be attempted in step S7.

[0095] In step S9 , following the judgment in step S8 that the preview confirmation is OK, the captured image is stored in the storage unit 62 , and the process proceeds to the end.

[0096] [About PSPNet structure( Figure 8 )]

[0097] One side reference Figure 8 On the one hand, the PSPNet structure selected as an example of a machine learning model in the pupil detection processing unit 632 is explained. PSPNet is an FCN (abbreviation for "Fully Convolution Network") that uses spatial pyramid pooling in the last layer to add 4-scale context features, thereby improving the accuracy of input images that require a wide range of context. In addition, "FCN" is a method of using CNN for semantic segmentation tasks, and refers to a model that does not use a fully convolutional layer but only consists of a convolutional layer.

[0098] like Figure 8 As shown, PSPNet80 has input image data 81, CNN82 (CNN is the abbreviation of “Convolutional Neural Network”), feature map 83, pyramid pooling module 84, convolution layer 85 and output prediction data 86.

[0099] CNN 82 is called a convolutional neural network, and is a network architecture for deep learning that directly learns from input image data 81, and generates a feature map 83. That is, when the input image data 81 is input to CNN 82, a feature map 83 (D channel) that collects a wide range of global surrounding information (wide range context) is generated for the feature vector of each pixel.

[0100] The pyramid pooling module 84 performs average pooling at four pyramid levels to generate an extended feature map 841. First, the entire image is pooled through grids of [6×6], [3×3], [2×2], and [1×1] to generate four pooled feature maps. Next, the four pooled feature maps are passed through a 1×1 convolution layer, and the dimension is reduced from the D channels of the original feature map 83 to D / 4 channels. Next, the four pooled feature maps are upsampled to align with the spatial size of the original feature map 83. Finally, the extended feature map 841 is generated by coupling the four pooled feature maps to the channels behind the original feature map 83. Here, "pooling" refers to the process of reducing the image size according to a rule. "Upsampling" refers to the process of increasing the spatial resolution of the feature map.

[0101] The convolution layer 85 obtains output prediction data 86 based on the expanded feature map 841 of the pyramid pooling module 84. That is, if the expanded feature map 841 is input to the convolution layer 85, classification recognition is performed for each pixel (pixel), and the final output prediction data 86 is obtained.

[0102] Therefore, if the input image data 81 is used as stereo camera image data and the feature map is used as the pupil area, the output prediction data 86 finally obtained is the model output pupil probability map. The model output pupil probability map is a map in which, when the stereo camera image data is subdivided in pixel units, a larger value is written in pixel units as the pupil probability level is higher in the area with high pupil probability. For example, Figure 8 In the output prediction data 86 (=model output pupil probability map) described in , the black background indicates the area other than the pupil where a low pupil probability value such as zero is written, and the blank area indicates the pupil prediction area where a high pupil probability value is written.

[0103] [Pupil detection processing structure ( Fig. 9 )]

[0104] One side reference Fig. 9 The process Figure 1 The pupil detection processing structure executed in the pupil detection processing unit 632 is described. In addition, the pupil detection processing is performed by serial processing of performing the pupil detection processing on the other camera image when the pupil detection processing on one camera image of the anterior ocular stereo camera image is completed. Fig. 9 . In addition, since both the right camera image data and the left camera image data are dynamic image data, a still camera image of one frame acquired at the start time of the pupil detection process is used as an input image, and the process is repeatedly executed at a predetermined control cycle (e.g., 8 msec). In addition, as the pupil detection process, the pupil detection process for the right camera image and the pupil detection process for the left camera image may be executed in parallel rather than in serial processing.

[0105] In step S401, after the start, an anterior ocular camera still image from a single camera of the anterior ocular stereo camera 22 is acquired, and as preprocessing, resizing and normalization to an input size matching the learned pupil area prediction model are applied. In resizing, for example, the size of the acquired anterior ocular camera still image is set to half the size to reduce the resolution.

[0106] In step S402, following the resizing / normalization in step S401, inference is performed by inputting the resized anterior ocular camera still image to the learned pupil area prediction model. Inference is performed using the anterior ocular camera still image and the learned pupil area prediction model, thereby obtaining a model output pupil probability map as model output information.

[0107] In step S403, following the acquisition of the model output pupil probability map in step S402, the pupil probability map is acquired by applying numerical conversion processing to the pupil probability values ​​written in the model output pupil probability map using an activation function. Here, the "activation function" refers to a nonlinear conversion processing function of a neural network, and a representative example is the S-type function (Sigmoid [Sigmoidal] function) that converts an arbitrary value into a value between 0.0 and 1.0. That is, since the numerical value of the pupil probability value is written without restriction into the model output pupil probability map output from the learned pupil area prediction model, the numerical width becomes wider. In contrast, if the numerical conversion processing is performed by the S-type function, the numerical width of the pupil probability value is converted to a narrow numerical width based on [0.0 to 1.0].

[0108] In step S404, following the acquisition of the pupil probability map in step S403, a pupil candidate area is acquired by threshold processing. Here, "threshold processing" refers to the extraction processing of retaining the area above the first threshold value (for example, about 0.6) of the pupil probability value as the pupil candidate area in the pupil probability map in which the pupil probability value based on the numerical width of [0.0 to 1.0] is written. In other words, it refers to the processing of deleting the area whose pupil probability value is less than the first threshold value from the pupil probability map. In addition, the threshold processing of extracting the pupil candidate area using the first threshold value is an example of the distribution recognition processing of the pupil probability value in the pupil probability map.

[0109] In step S405 , following the acquisition of the pupil candidate region in step S404 , the acquired pupil candidate region is labeled by adding a label.

[0110] In step S406, following the marking in step S405, it is determined whether the number of tags is plural. If yes (plural), the process proceeds to step S408, and if no (one), the process proceeds to step S407.

[0111] In step S407, following the judgment of "No (one)" in step S406, a pupil label is assigned to a pupil candidate area with a label. That is, by assigning a pupil label to a pupil candidate area, the pupil candidate area assigned with the pupil label is detected as the pupil area. In step S408, following the judgment of "Yes (multiple)" in step S406, a pupil label is assigned to the pupil candidate area with the largest label (largest area) among multiple pupil candidate areas with labels. That is, by assigning a pupil label to the pupil candidate area with the largest area, the pupil candidate area assigned with the pupil label is detected as the pupil area. In addition, in the case of multiple pupil candidate areas, the pupil candidate area with the largest area is an example of a method for determining a candidate area with the highest pupil possibility.

[0112] In step S409, following the detection of the pupil region to which the pupil label is assigned in step S407 or step S408, ellipse approximation is applied to the detected pupil region to obtain pupil center coordinates (XY coordinates) indicating the position of the center point of the ellipse. In addition, the pupil center coordinates obtained by applying the ellipse approximation are an example of pupil position coordinates based on the shape of the detected pupil region.

[0113] In step S410, following the acquisition of the pupil center coordinates in step S409, it is determined whether the area of ​​the detected pupil region is within the set area range (first pupil determination condition). In the case of OK (the first pupil determination condition is satisfied), the process proceeds to step S411, and in the case of NG (the first pupil determination condition is not satisfied), the process proceeds to step S412. Here, as the set area range, for example, the upper limit area is set to a pupil area based on a diameter (9.6 mm) that is 1.2 times the pupil diameter of 8 mm, which is considered large, and the lower limit area is set to a pupil area based on a diameter (1 mm) that is 0.5 times the pupil diameter of 2 mm, which is considered small.

[0114] In step S411, following the OK judgment in step S410, based on the pupil probability map, it is determined whether the occupancy ratio of pupil probability values ​​above the second threshold value higher than the first threshold value is above the set ratio relative to the entire detected pupil area (second pupil determination condition). In the case of OK (the second pupil determination condition is satisfied), the process proceeds to step S413, and in the case of NG (the second pupil determination condition is not satisfied), the process proceeds to step S412. Here, the "second threshold value" is set to a value (for example, about 0.8) higher than the first threshold value (for example, about 0.6). The "set ratio" is an occupancy ratio set to a high value (for example, about 70% to 80%) based on experimental data of occupancy ratios when the detected pupil area is a true pupil area. In addition, the second pupil determination condition in step S411 is an example of a condition for confirming the aggregation of pupil probability values ​​above the threshold value in the detected pupil area with the help of the pupil probability map.

[0115] In step S412, if at least one of the first pupil determination condition in step S410 and the second pupil determination condition in step S411 is not satisfied, an error is set indicating that pupil area detection has failed, and the process proceeds to the end.

[0116] In step S413, following the OK judgment in step S411, the camera still image is resized to return to the original size, and if both the first pupil determination condition and the second pupil determination condition are satisfied, the detection processing result is output as successful pupil area detection, and the process proceeds to the end. Here, the "detection processing result" refers to the pupil center coordinates output by the automatic alignment unit 633.

[0117] [Background Technology and Technical Problems ( Fig.10 , Figures 11A to 11D )]

[0118] In the case of the ophthalmic device A, when the eye characteristics of the examined eye E are examined or photographed with the subject supporting the lower jaw on the jaw rest 30, the position of the examined eye E shifts each time the subject changes, and therefore it is necessary to adjust the alignment control of the relative position relationship between the examined eye E and the main body 20.

[0119] However, the stereo camera image captured by the anterior ocular stereo camera 22 is prone to vignetting due to the pupil being blocked by the reflection of the eyelids or eyelashes. In particular, in the case of an ophthalmic device A having a fundus camera and OCT with a wide-angle shooting angle, the lens diameter of the objective lens 51 becomes larger due to the optical constraints accompanying the widening of the shooting angle. Therefore, the camera installation angle of the anterior ocular stereo camera 22 composed of the right camera 22a and the left camera 22b arranged on both sides of the objective lens 51 becomes an acute angle compared to the camera installation angle in the case of a small lens diameter (see Figure 2 as well as Figure 5 ). Therefore, the pupil shape of the eye under examination E becomes an elongated ellipse, and vignetting, in which the pupil is blocked by the reflection of the eyelids or eyelashes, etc., is more likely to occur. Figure 5 The right camera 22a and the left camera 22b of the front eye stereo camera 22 shown should be as follows Figure 2 Although the arrangement is shown in a left-right positional relationship, for convenience, the arrangement is shown in an up-down positional relationship.

[0120] In contrast, in the background art, the pupil of the eye to be examined E is detected based on a binary image acquired by image processing in which the pupil area is recognized from a stereo camera image captured by the anterior ocular portion. Fig.10 The process Figure 1 The pupil detection processing structure in the background technology is described. Fig.10 Flowchart and Fig. 9 Similarly, in the stereo camera image of the anterior ocular part, pupil detection processing for the right camera image and pupil detection processing for the left camera image are performed by serial processing. In addition, since both the right camera image data and the left camera image data are dynamic image data, a still camera image of one frame acquired at the start time of the pupil detection processing is used as an input image and is repeatedly executed in a predetermined control cycle.

[0121] In step S501, after the start, a 1 / 4 image is generated by cutting the anterior ocular still camera image from the stereo camera. In the next step S502, a bright spot (= alignment bright spot) projected from the alignment optical system 521 to the anterior ocular segment Ea of the subject's eye E is searched from the 1 / 4 image. In step S503, it is determined whether there is an alignment bright spot in the 1 / 4 image. If there is a bright spot, the process proceeds from step S503 to step S504, and if there is no bright spot, the process proceeds from step S503 to step S505.

[0122] In step S504, a portion of the image cut out around the bright spot is binarized. In step S505, the entire image is binarized. Here, "binarization" means that the pupil area is extracted as the target, and an image is formed into two images divided into a low brightness area and a high brightness area according to a brightness threshold set to a low brightness.

[0123] In step S506, following the binarization in step S504 or step S505, the low brightness area (one or more) extracted by the binarization is marked. In the next step S507, in the low brightness area extracted by the binarization, the small area of ​​low brightness below the set pupil area threshold is deleted.

[0124] In step S508, the number of labels n of the low-brightness regions remaining after the small low-brightness regions are deleted is determined. If the number of labels n is n=0, the process proceeds to step S509; if the number of labels n is n>1, the process proceeds to step S510; if the number of labels n is n=1, the process proceeds to step S511. In step S509, following the determination of n=0 in step S508, the process proceeds to the end as an error setting indicating that the pupil region detection failed.

[0125] In step S510, following the determination of n>0 in step S508, the label with the largest roundness among the plurality of retained low-brightness regions is selected. Here, "roundness" refers to an index value indicating the roundness calculated based on the relationship between the area and the perimeter of the region, and is expressed by roundness = 4πS / L 2 (S is the area, L is the perimeter). In step S511, following the judgment that n=1 in step S508, the low-brightness area judged to be n=1 is acquired as a pupil candidate. Alternatively, following the label selection in step S510, the selected low-brightness area is acquired as a pupil candidate. In the next step S512, following the acquisition of the pupil candidate in step S511, an elliptical approximation is applied to the contour edge of the pupil candidate to acquire the pupil center coordinates (XY coordinates).

[0126] In step S513, following the ellipse approximation of the contour edge in step S512, it is determined whether the pupil determination condition based on shape is satisfied. In the case of OK (the pupil determination condition is satisfied), the detection processing result (pupil center coordinates) is output as a successful pupil area detection, and the process proceeds to the end. On the other hand, in the case of NG (the pupil determination condition is not satisfied), the process proceeds to step S509 as an error setting indicating that the pupil area detection has failed, and the process proceeds to the end. Here, the "pupil determination condition based on shape" refers to the roundness condition that the roundness calculation value of the pupil candidate is above the roundness determination threshold and the area area condition that the area area calculation value of the pupil candidate is above the area determination threshold. Therefore, of the roundness condition and the area area condition of the pupil candidate, the pupil detection is determined to be successful only when both conditions are satisfied. If at least one of the two conditions is not satisfied, the pupil detection is determined to be a failure.

[0127] Next, refer to Figures 11A to 11D While explaining the technical problems of pupil detection in the background technology. In addition, in the background technology, such as Fig.11A As shown, the input information is set to be a stereo camera image of the eyelid LI and the eyelashes LA reflected in the upper part of the pupil P.

[0128] The method for detecting the pupil area in the background art is as follows Fig.10 The flowchart shown in Figure 1 is as follows. Fig.11A The stereo camera image shown in FIG. 1 is searched to see whether a bright spot BS projected onto the anterior eye Ea can be detected. Then, if the bright spot BS is not found, the entire image is binarized using a brightness threshold for identifying the pupil P with the lowest brightness in the image, as shown in FIG. Fig. 11B As shown, if the bright spot BS is found, the image around the bright spot is binarized using a brightness threshold for identifying the pupil P with the lowest brightness in the image. Then, a method of identifying and detecting the pupil area from the acquired binary image is adopted.

[0129] Therefore, in the case of background technology using stereo camera images to detect pupil areas, such as Fig. 11C As shown in FIG. 1 , if the eyelid LI or eyelashes LA having a brightness higher than the pupil P is reflected, a part of the pupil region is lost by binarization, and an image of a flattened pupil region shape PS is obtained. In this way, if there is a portion having a brightness level higher than the pupil brightness in the pupil image due to the reflection of the eyelid LI or eyelashes LA, the portion having a higher brightness is not recognized as the pupil region, and the roundness or area of ​​the recognized pupil region is reduced.

[0130] Therefore, if part of the pupil region is lost due to binarization, and it is determined that the area of ​​the pupil region shape PS is small, then Fig.10In step S507 of the flowchart of FIG. 1 , a small low-brightness area is deleted. If the number of labels n is determined to be n=0 by the deletion in step S508 , the process proceeds to step S509 , in which an error is set as indicating that pupil area detection has failed.

[0131] On the other hand, if it is determined that the area of ​​the pupil region shape PS is not small even if part of the pupil region is missing due to binarization, then Fig.10 In step S507 of the flowchart, the low brightness area is not deleted. Then, if it is determined in step S508 that the number of tags n is n>1, the process proceeds to step S510→step S511→step S512. In addition, if it is determined in step S508 that the number of tags n is n=1, the process proceeds to step S511→step S512. In step S511, the low brightness area is obtained as a pupil candidate, and in step S512, if Fig.11D As shown, the ellipse approximation EA is applied to the contour edge of the pupil candidate to obtain the pupil center coordinates PC.

[0132] However, in step S513, the next step after step S512, it is determined whether the pupil determination conditions (roundness condition, area condition) based on the shape are satisfied. Fig. 11C If the roundness of the pupil candidate or the area of ​​the pupil candidate is reduced due to the reflection, the pupil determination condition based on the shape will not be satisfied. Then, in step S513, if at least one of the two conditions of the roundness condition and the area condition is not satisfied, the process proceeds to step S509, and in step S509, an error is set as indicating that the pupil area detection has failed.

[0133] Thus, in the case of the background technology of detecting the pupil of the inspected eye E by image processing of a stereo camera image that recognizes and photographs the anterior ocular part, if the roundness or area of ​​the pupil area is reduced due to the reflection of the eyelid LI or the eyelashes LA, it is difficult to detect the pupil area correctly and stably. In addition, for diseased eyes such as cataracts or mydriasis, the roundness of the low-brightness area based on binarization is reduced or the area of ​​the low-brightness area based on binarization is reduced for the original pupil area, making it difficult to detect the pupil area correctly and stably. Furthermore, for the automatic alignment of the pupil, since it is performed on the condition that the pupil detection from two directions is successful for the photographing eye, it directly reflects the technical problem that the detection of the pupil area lacks stability, and the stability of the automatic alignment of the pupil is reduced.

[0134] [Pupil detection function ( Fig. 9 , Figures 12A to 12D )]

[0135] In response to the technical problems of pupil detection in the above-mentioned background technology, we focus on the following point, that is, if AI image recognition technology using a learned model is used for the pupil detection method in an ophthalmic device, the influence that hinders pupil detection in the background technology method will be eliminated, thereby enabling the pupil area to be detected correctly and stably.

[0136] That is, the ophthalmic apparatus A includes a learning model setting unit 621 that sets a learning model related to anterior ocular images of the inspected eye E, and a pupil detection processing unit 632 that detects a pupil area of ​​the inspected eye E. The learning model setting unit 621 prepares a plurality of teacher data in which pupil area information is added to pre-collected anterior ocular camera image data, and sets a learning pupil area prediction model generated by causing a selected machine learning model to read the teacher data for learning processing. The pupil detection processing unit 632 is configured to detect the pupil area PA of the inspected eye E based on the prediction information of the pupil area output by the model through inference execution, in which the anterior ocular camera image data acquired from the anterior ocular stereo camera 22 is input to the learning pupil area prediction model.

[0137] In this way, the pupil area detection information is not set to the front eye camera image information that is affected by the reflection that hinders pupil detection, but is set to the prediction information of the pupil area output by the model through inference execution from the learned pupil area prediction model. Therefore, the prediction information of the pupil area output by the model is information that accurately predicts the original pupil area where the pupil P is located, regardless of the influence of the reflection that hinders pupil detection. Therefore, even if there is a portion in the pupil image with a brightness level higher than the pupil brightness due to the reflection of the eyelid LI or the eyelashes LA, the original area where the pupil P is located, including the portion with higher brightness, is recognized as the pupil area (refer to Fig. 12C Therefore, during pupil detection, AI image recognition using the learning completion model can robustly detect the pupil area PA.

[0138] Next, refer to Fig. 9 as well as Figures 12A to 12D The pupil detection processing function in Example 1 is described. In addition, in Example 1, in order to be able to compare with the background technology, as shown in FIG. Fig. 12A As shown, the front ocular camera still image input to the learning pupil area prediction model is set to the background technology ( Fig.11A ) the same image.

[0139] In the pupil area detection process in Example 1, resizing and normalization are applied as preprocessing to the anterior ocular camera still image obtained in step S401. In the next step S402, the model output pupil probability map is obtained by performing inference by inputting the resized anterior ocular right camera still image into the learned pupil area prediction model. In the next step S403, the pupil probability map M is obtained by performing numerical conversion processing on the pupil probability values ​​written into the model output pupil probability map using an activation function. For example, Fig. 12B As shown, although the eyelid LI and the eyelashes LA are reflected in the upper part of the pupil P, the pupil probability map M is a map showing that the pupil probability value is high in the original pupil area as a whole that is not reflected. Fig. 12B The pupil probability map M shown is a map in which pupil probability values ​​are expressed as a black-and-white image with shades.

[0140] In the next step S404, the pupil candidate area is obtained by threshold processing. In the next step S405, the obtained pupil candidate area is marked with a label. In the next step S406, it is determined whether the number of labels is multiple. In the case of no (1), the process proceeds to step S407, and a pupil label is assigned to a pupil candidate area with a label. On the other hand, in the case of yes (multiple), the process proceeds to step S408, and among multiple pupil candidate areas with labels, a pupil label is assigned to the pupil candidate area with the largest label. The pupil candidate area to which the pupil label is assigned in step S407 or step S408 is detected as the pupil area. For example, Fig. 12C As shown, the pupil region PA assigned with the pupil label substantially matches the regional shape of the pupil P based on the inspected eye E, regardless of the eyelid LI and the eyelashes LA reflected in the upper part of the pupil P. That is, the pupil region PA assigned with the pupil label obtained by the pupil detection processing in the first embodiment has a highly rounded and large regional shape.

[0141] In the next step S409, elliptical approximation is applied to the pupil area PA assigned with the pupil label to obtain the pupil center coordinates PC (XY coordinates). Fig.12D As shown in FIG. 1 , the pupil center coordinates PC are obtained as the position coordinates of the center point of the ellipse by applying the ellipse approximation EA to the pupil area PA to which the pupil label is assigned.

[0142] In the next step S410, it is determined whether the area of ​​the pupil region PA assigned with the pupil label is within the set area range, which is the first pupil determination condition. If it is determined to be NG (the first pupil determination condition is not met), the process proceeds to step S412 as an error setting indicating that the pupil region detection has failed, and the process proceeds to the end. If it is determined to be OK (the first pupil determination condition is met), the process proceeds to step S411. Here, the first pupil determination condition is an area confirmation condition that excludes the situation where the area of ​​the pupil region PA assigned with the pupil label becomes a small area that is actually impossible to exist or a large area due to misdetection or the like. Therefore, the first pupil determination condition is determined to be OK as long as the pupil region PA is predicted based on the pupil probability map M obtained normally.

[0143] In the next step S411, the second pupil determination condition is judged whether the ratio of the area with a pupil probability value above the second threshold higher than the first threshold to the pupil candidate area above the first threshold of the pupil probability map M is above the pupil determination ratio threshold. If it is judged as NG (the second pupil determination condition is not satisfied), the process proceeds to step S412 as an error setting indicating that the pupil area detection failed, and the process proceeds to the end. If it is judged as OK (the first pupil determination condition is satisfied), the process proceeds to step S413, the camera still image is resized to return to the original size, and the detection processing result is output as a successful pupil area detection, and the process proceeds to the end. Here, the second pupil determination condition is a prediction accuracy confirmation condition that confirms that the prediction accuracy of the detected pupil area PA is high based on the distribution of the pupil probability value of the pupil area PA predicted based on the pupil probability map. Therefore, the second pupil determination condition is judged as OK as long as the pupil area PA is predicted based on the pupil probability map M acquired normally.

[0144] Thus, in the case of Example 1 in which pupil area PA is detected based on the model output pupil probability map, even if there is a reflection of eyelid LI or eyelash LA in the input front ocular camera still image, the pupil area will not be affected by the reflection, so that the pupil area can be detected correctly and stably. That is, by adopting AI image recognition using a learned pupil area prediction model that can eliminate the influence that hinders pupil detection as a detection method for pupil area PA, accuracy and stability in pupil area detection are obtained. In particular, for the predicted pupil area PA, the accuracy in pupil area detection is improved by including the satisfaction of the first pupil determination condition (region area confirmation condition) and the satisfaction of the second pupil determination condition (prediction accuracy confirmation condition) as detection success conditions for the pupil area PA.

[0145] In addition, even for diseased eyes such as cataracts or miosis, by adopting AI image recognition using a learning-completed pupil area prediction model, it will not be affected by the shape flattening of the low-brightness area or the narrowing of the area of ​​the low-brightness area as in the background technology. Therefore, the pupil area PA can be detected correctly and stably. Furthermore, for the automatic alignment of the pupil P, although the condition is that the pupil detection from two directions for the photographing eye is successful, the advantage of the pupil detection processing that can stably detect the pupil area PA from two directions is reflected, and the stability of the automatic alignment of the pupil P is improved.

[0146] As described above, the ophthalmologic apparatus A according to the first embodiment can achieve the following effects.

[0147] (1) The ophthalmic apparatus A has an anterior ocular camera (anterior ocular stereo camera 22) for photographing the anterior ocular segment Ea of the subject's eye E. It also has a learned model setting unit 621 for setting a learned model related to the anterior ocular image of the subject's eye E, and a pupil detection processing unit 632 for detecting the pupil area PA of the subject's eye E. The learned model setting unit 621 sets a learned pupil area prediction model, which is generated by preparing a plurality of teacher data in which pupil area information is added to pre-collected anterior ocular camera image data, and by a learning process in which the teacher data is read into a selected machine learning model. The pupil detection processing unit 632 detects the pupil area PA of the subject's eye E based on the predicted information of the pupil area output by the model through inference execution in which the anterior ocular camera image data acquired from the anterior ocular camera (anterior ocular stereo camera 22) is input into the learned pupil area prediction model.

[0148] Therefore, during pupil detection, AI image recognition using the learned model can robustly detect the pupil area PA.

[0149] (2) The learned model setting unit 621 sets a learned pupil area prediction model, which collects multiple anterior ocular camera image data regardless of the type or type of ophthalmic device, adds pupil area information to the anterior ocular camera image data to generate teacher data, and uses the teacher data to generate the model.

[0150] Therefore, by setting a learned general model as a learned pupil area prediction model, it is possible to detect the pupil area PA by performing inference that captures the features of the input anterior segment camera image data regardless of the type or type of ophthalmic apparatus. Here, the "features of the anterior segment camera image data" refer to the shape features of the subject's eye E, eyelid LI, eyelashes LA, etc., which are reflected in the frame as the anterior segment camera image, and the features of the positional relationship of each part with respect to the entire frame.

[0151] (3) The pupil detection processing unit 632 performs inference execution on the anterior ocular camera image data input by the model, and obtains a pupil probability map M in which pupil probability values ​​are written in pixel units into the anterior ocular camera image data. A high probability value area extracted by distribution recognition processing (binarization threshold processing) of the pupil probability values ​​in the pupil probability map M is detected as a pupil candidate area.

[0152] Therefore, when detecting a pupil candidate region, by extracting the distribution recognition processing of the pupil probability value of the high probability region from the pupil probability map M acquired by inference execution, it is possible to detect an appropriate region with a high pupil probability value as a pupil candidate region.

[0153] (4) When the pupil detection processing unit 632 detects multiple pupil candidate areas, it labels each candidate area, assigns a pupil label to the candidate area with the highest pupil possibility (the candidate area with the largest area), and detects the pupil candidate area assigned with the pupil label as the pupil area PA.

[0154] Therefore, when a plurality of pupil candidate regions are detected, a pupil label is assigned to a candidate region with the highest pupil probability, and a suitable region can be detected as the pupil region PA by selecting the pupil candidate region assigned the pupil label.

[0155] (5) When the pupil area PA is detected, the pupil detection processing unit 632 acquires pupil position coordinates (pupil center coordinates PC) based on the shape of the detected pupil area PA.

[0156] Therefore, when acquiring pupil position coordinates (pupil center coordinates PC), the pupil position coordinates output as pupil position information can be acquired by processing to identify the pupil position based on the shape of the detected pupil area PA.

[0157] (6) The pupil detection processing unit 632 determines the first pupil determination condition and the second pupil determination condition. The first pupil determination condition is to confirm that the area of ​​the detected pupil region PA is within a set area range, and the second pupil determination condition is to confirm the aggregation of pupil probability values ​​above a threshold value in the detected pupil region PA with the help of the pupil probability map M. If at least one of the first pupil determination condition and the second pupil determination condition is not satisfied, an error setting indicating that the pupil region detection has failed is made. If both the first pupil determination condition and the second pupil determination condition are satisfied, a detection processing result indicating that the pupil region detection has been successful is output.

[0158] Therefore, when detecting the pupil area PA, by determining that the area size confirmation condition and the prediction accuracy confirmation condition are satisfied as the detection success condition of the pupil area PA, the reliability of the detection of the pupil area PA can be improved.

[0159] (7) A main body 20 having a built-in optical system, a driving unit (XYZ driving unit 18) for moving the main body 20 in three-dimensional directions relative to the support unit 10, and a control unit 60 for controlling each part of the device. The anterior ocular camera (anterior ocular stereo camera 22) is composed of at least two cameras (right camera 22a, left camera 22b) arranged at the outer peripheral position of the objective lens 51 of the main body 20 and arranged so that the lens optical axis is tilted toward the anterior ocular segment Ea of the subject's eye E. In the control unit 60, an alignment control unit 63 that controls the adjustment of the relative positional relationship between the subject's eye E and the main body 20 includes a pupil detection processing unit 632 and an automatic alignment unit 633 that performs automatic alignment of the pupil P. The pupil detection processing unit 632 detects the pupil area using the camera image from the anterior ocular camera and the learned pupil area prediction model set by the learned model setting unit 621. When the automatic alignment unit 633 automatically aligns the pupil P, if the detection of the pupil area PA of the examined eye E is successful, the two pupil position coordinates (pupil center coordinates PC) are acquired, and a movement instruction for converging the three-dimensional current coordinates of the pupil P calculated based on the two pupil position coordinates to the three-dimensional target coordinates of the pupil P is output to the drive unit (XYZ drive unit 18).

[0160] Therefore, the advantage of the pupil detection process that the pupil area PA can be stably detected even when using an image from an anterior ocular camera where reflection is likely to occur during automatic alignment of the pupil P is reflected, and the stability of the automatic alignment can be improved.

[0161] [Example 2]

[0162] Embodiment 2 is an example in which an anterior ocular segment three-direction camera 22 ′ is used as the anterior ocular segment camera, compared to Embodiment 1 which uses the anterior ocular segment stereo camera 22 .

[0163] like Fig.13 As shown, the anterior ocular 3-direction camera 22' has a right camera 22a, a left camera 22b, and a lower camera 22c arranged in an objective lens unit housing 71 as a structural component of the objective lens unit 70. Furthermore, the objective lens unit 70 has a right camera shooting light 25a, a left camera shooting light 25b, and a lower camera shooting light 25c as a camera shooting light 25 attached to the anterior ocular 3-direction camera 22'. Here, the objective lens unit 70 refers to a unit composed of the anterior ocular 3-direction camera 22' and the camera shooting light 25 integrally assembled on the objective lens 51 of the optical system 50.

[0164] The front ocular three-direction camera 22' is located at the periphery of the objective lens 51. When a horizontal dividing line HL is drawn that divides the lens into two parts in the vertical direction through the optical axis OA of the objective lens 51, three cameras 22a, 22b, and 22c are arranged in the lower area of ​​the horizontal dividing line HL. The camera shooting light 25 has a right camera shooting light 25a at the upper position of the right camera 22a, a left camera shooting light 25b at the upper position of the left camera 22b, and two lower camera shooting lights 25c at the two sides of the lower camera 22c.

[0165] Here, a small camera having a lens unit and an image sensor is used as the anterior ocular segment three-direction camera 22'. In addition, an infrared LED (abbreviation of "Light Emitting Diode") is used as the camera shooting light 25, which irradiates infrared light to the subject's eye E to increase the brightness difference between the pupil P and other parts in the captured anterior ocular segment image.

[0166] When a vertical dividing line VL is drawn that passes through the optical axis OA of the objective lens 51 and divides the lens into two parts in the left-right direction, the right camera 22a and the left camera 22b are arranged at the left and right sides of the vertical dividing line VL that are line-symmetrical. The lower camera 22c is arranged at a line position of the vertical dividing line VL. In addition, the right camera 22a, the left camera 22b, and the lower camera 22c are arranged to have an upward tilt angle so that the optical axis of each camera lens is directed toward the pupil P of the eye E to be examined.

[0167] The automatic alignment of the pupil P is described. In the case of Example 2, since the anterior ocular three-directional camera 22' is used, the camera image data from the three cameras of the right camera 22a, the left camera 22b, and the lower camera 22c are obtained. Therefore, when the pupil detection is successful in all the camera image data of the three cameras 22a, 22b, and 22c, when the pupil P is automatically aligned, two cameras are automatically selected, and the three-dimensional current coordinates of the pupil P are calculated based on the pupil position coordinates from the two selected cameras. Here, the method of automatically selecting two cameras is, for example, to use the height of the roundness or the height of the probability value as the selection criterion, exclude the one judged to be the lowest among the three, and retain the higher two.

[0168] In the case of the second embodiment, since the anterior ocular three-direction camera 22' is used, even if the pupil detection fails using the image data of one of the three cameras 22a, 22b, and 22c, the failure of the pupil detection is allowed. That is, if the pupil detection succeeds using the image data of two cameras, the three-dimensional current coordinates of the pupil P can be calculated based on the two pupil position coordinates when the pupil P is automatically aligned, as in the first embodiment. In addition, since other structures and effects are the same as those in the first embodiment, illustration and description are omitted.

[0169] The ophthalmic device of the present disclosure has been described above based on Embodiment 1 and Embodiment 2. However, the specific configuration is not limited to these embodiments, and design changes and additions are permitted without departing from the gist of the invention of each claim.

[0170] In Example 1, as an anterior ocular camera, an example of an anterior ocular stereo camera 22 composed of a right camera 22a and a left camera 22b that perform anterior ocular photography of one photographing eye from the left and right directions in a manner suitable for automatic alignment of the pupil P is shown. In Example 2, an example of an anterior ocular three-direction camera 22' composed of a right camera 22a, a left camera 22b, and a lower camera 22c that perform anterior ocular photography of one photographing eye from the left and right directions and the lower direction is shown. However, the anterior ocular camera is not limited to an anterior ocular stereo camera or an anterior ocular three-direction camera. For example, it can also be a structure composed of more than four cameras arranged at the peripheral position of the objective lens possessed by the main body. Further, in the case where the pupil detection processing of the present disclosure is applied to various purposes other than automatic alignment, in order to be suitable for each purpose, for example, it can also be a separate anterior ocular camera that performs anterior ocular photography of one photographing eye from one direction. In addition, it can also be an anterior ocular camera that simultaneously photographs the anterior right eye and the anterior left eye of the photographing eyes of the right eye and the left eye.

[0171] In Example 1, a built-in example in which the learning completion model setting unit 621 is set in the storage unit 62 of the control unit 60 of the ophthalmic device A is shown. However, the location where the learning completion model setting unit is set is not limited to the built-in example of the control unit of the ophthalmic device. For example, even the built-in example may be an example in which a storage unit dedicated to the learning completion model setting for storing and setting the learning completion model is set in a location other than the control unit of the ophthalmic device. In short, it can be set in a location that can process the automatic alignment of the pupil in real time and can perform edge processing based on edge calculation. Therefore, as long as it is a location that can satisfy edge processing, for example, an external memory connected to an external memory connection terminal can also be used as the learning completion model setting unit.

[0172] In the first embodiment, as the learning model setting unit 621, an example is shown in which a learning general model is set by using teacher data generated by adding pupil area information to a plurality of anterior ocular stereo camera image data collected regardless of the type or type of ophthalmic device. However, as the learning model setting unit, it is also possible to use an example in which the number of anterior ocular camera images collected is distinguished according to the difference in the configuration of the anterior ocular camera, pupil area information is added to the anterior ocular camera image data to generate teacher data, and then, the learning pupil area prediction model generated for different ophthalmic devices or different camera configurations is set using the teacher data. In this case, when detecting the pupil area, it is possible to cope with the fact that the features of the anterior ocular camera image data are different due to the difference in the anterior ocular camera configuration, and the pupil area can be accurately detected regardless of the difference in the camera configuration. In addition, it is also possible to use an example in which the camera configuration is roughly classified according to the system, and the learning pupil area prediction model generated for each classified camera configuration system is set.

[0173] In Example 1, an example is shown in which PSPNet is selected as a machine learning model, and a learning completed pupil area prediction model generated by causing PSPNet to read a learning process of a plurality of prepared teacher data is set as a learning completed model setting unit 621. However, the machine learning model is not limited to being selected as PSPNet. For example, UNet may also be selected as a machine learning model. In addition, "UNet" is one of the FCNs, and is a convolutional neural network-based model developed for biomedical image segmentation. In UNet, when a feature map is enlarged for processing on the decoder side, it is cut and used in such a way that the feature map on the encoder side is the same size. Furthermore, as a machine learning model, a model other than PSPNet and UNet may also be selected. In particular, a machine learning model having a convolutional neural network structure (CNN structure) is preferred, which is a network architecture for deep learning for image recognition that is good at identifying feature parts from camera images.

[0174] In the first embodiment, the pupil detection processing unit 632 shows an example of detecting a high probability value region extracted by threshold processing of binarizing the pupil probability value in the pupil probability map M using a first threshold value as a pupil candidate region. However, as long as the pupil detection processing unit detects a high probability value region extracted by distribution recognition processing of the pupil probability value in the pupil probability map as an example of a pupil candidate region, it is not limited to the example of performing binarization threshold processing. For example, it may be an example of extracting a plurality of pupil candidate regions using a plurality of threshold values, comprehensively performing distribution analysis of pupil probability values ​​on the plurality of pupil candidate regions, and extracting the pupil candidate regions.

[0175] In the first embodiment, as the pupil detection processing unit 632, an example is shown in which, when multiple pupil candidate regions are detected, each candidate region is marked separately, and a pupil label is assigned to the candidate region with the largest pupil candidate region area. However, as long as the pupil detection processing unit can assign a pupil label to the candidate region with the highest pupil probability, it is not limited to the example of assigning a pupil label to the candidate region with the largest pupil candidate region area. For example, the example of assigning a pupil label to the candidate region with the highest average value of the pupil probability value output by the model among the pupil candidate regions may be used, and the example of assigning a pupil label to the candidate region with the maximum value of the pupil probability value output by the model among the pupil candidate regions may also be used.

[0176] In the first embodiment, as the pupil detection processing unit 632, an example is shown in which, if the pupil area PA is detected, the ellipse approximation EA is applied to the shape of the detected pupil area PA, and the pupil center coordinates PC (the center coordinates of the approximate ellipse) are obtained. However, as long as the pupil position coordinates can be obtained based on the shape of the pupil area, the pupil detection processing unit is not limited to the example of obtaining the pupil center coordinates. For example, an example may be provided in which the pupil centroid coordinates (the coordinates of the position where the shape is kept balanced) are obtained based on the shape of the pupil area.

[0177] In the first embodiment, as the pupil detection processing unit 632, an example is shown in which the occupancy ratio of pupil probability values ​​above a second threshold value higher than the first threshold value relative to the entire pupil area PA is equal to or greater than a set ratio as the second pupil determination condition. However, as long as the pupil detection processing unit sets the second pupil determination condition as an example of a condition for confirming the aggregation of pupil probability values ​​above the threshold value in the detected pupil area using a pupil probability map, it is not limited to the example of the first embodiment. For example, the second pupil determination condition may be set to find the average value of the pupil probability values ​​of the entire pupil area, and the average value may be equal to or greater than the aggregation determination value indicating the aggregation of high pupil probability values, or the second pupil determination condition may be set to find the distribution of the pupil probability values ​​of the entire pupil area, and determine that the distribution form is a form indicating the aggregation of high pupil probability values.

[0178] In the first embodiment, as the automatic alignment unit 633, an example is shown in which, when the pupil P is automatically aligned, if the detection of the pupil area PA of the subject's eye E from two directions is successful, the three-dimensional current coordinates of the pupil P are calculated based on two pupil center coordinates PC. In the second embodiment, an example is shown in which, if the detection of the pupil area PA of the subject's eye E from three directions is successful, the three-dimensional current coordinates of the pupil P are calculated based on two automatically selected pupil center coordinates PC. However, as an automatic alignment unit, it is not limited to the successful detection of the pupil area from two directions or three directions. That is, since pupil position information from at least two directions is required to calculate the three-dimensional current coordinates, it is also possible to automatically select two of the cameras in the successful directions when the pupil detection is successful using camera images from four or more directions.

[0179] In Example 1, an ophthalmic device that uses a pupil detection processing technology based on AI image recognition to perform automatic alignment of a pupil P is shown, and an example of application to an ophthalmic device having a wide-angle fundus camera and OCT is shown. However, the application of an ophthalmic device using the pupil detection processing technology disclosed in the present invention is not limited to the application example to an ophthalmic device having a wide-angle fundus camera and OCT. For example, as long as it is an ophthalmic device that needs to perform automatic alignment of the pupil before observing the subject eye or before the ophthalmic characteristics of the subject eye are examined, it can also be applied to various ophthalmic devices other than Example 1.

[0180] In the pupil detection processing technology disclosed in the present invention, since the anterior ocular camera image data (dynamic image data) is used as the input image, the pupil area is repeatedly detected with a predetermined control cycle. Therefore, if the acquired pupil position coordinates are plotted along the time axis, the line of sight movement trajectory of the examined eye (eye tracking) is obtained. Therefore, for example, the pupil detection processing technology disclosed in the present invention is used for the detection of line of sight movement, and can be applied to the following ophthalmic device, that is, the ophthalmic device performs a line of sight movement inspection of how the pupil position of the examined eye (left and right eyes) moves when the visual mark that the examinee is looking at is moved in a predetermined manner. In addition, for example, the pupil detection processing technology disclosed in the present invention is used for the detection of the line of sight direction of the examinee, and can be applied to the following ophthalmic device, that is, the examinee wears a head-mounted display, and various conscious inspections are performed with the help of the displayed conscious inspection screen while monitoring the line of sight direction of the examinee.

Claims

1. An ophthalmic device having an anterior ocular camera for photographing the anterior ocular segment of an eye to be examined, characterized in that: The device comprises: a learning completed model setting unit for setting a learning completed model related to the anterior ocular segment image of the eye to be examined; and a pupil detection processing unit for detecting a pupil area of ​​the inspected eye, The learned model setting unit prepares a plurality of teacher data to which pupil area information is added to pre-collected anterior ocular camera image data, and sets a learned pupil area prediction model generated by causing a selected machine learning model to read the learning process of the teacher data. The pupil detection processing section detects the pupil area of ​​the eye to be examined based on prediction information of the pupil area output by a model by inputting anterior segment camera image data acquired from the anterior segment camera into inference execution of the learned pupil area prediction model.

2. The ophthalmic device according to claim 1, wherein: The learned model setting unit collects a plurality of the anterior ocular camera image data regardless of the type and type of the ophthalmic device, adds the pupil area information to the anterior ocular camera image data to generate the teacher data, and sets the learned pupil area prediction model generated using the teacher data.

3. The ophthalmic device according to claim 2, wherein: The pupil detection processing unit inputs the anterior ocular camera image data into the inference execution of the learned pupil area prediction model, obtains a pupil probability map in which pupil probability values ​​are written in pixel units in the anterior ocular camera image data, and detects high probability value areas extracted by distribution recognition processing of the pupil probability values ​​in the pupil probability map as pupil candidate areas.

4. The ophthalmic device according to claim 3, wherein: When the pupil detection processing unit detects a plurality of pupil candidate regions, it labels each candidate region, assigns a pupil label to a candidate region with the highest pupil probability, and detects the pupil candidate region assigned with the pupil label as a pupil region.

5. The ophthalmic device according to claim 4, characterized in that When the pupil region is detected, the pupil detection processing unit acquires pupil position coordinates based on the shape of the detected pupil region.

6. The ophthalmic device according to claim 5, characterized in that The pupil detection processing unit determines a first pupil determination condition and a second pupil determination condition, wherein the first pupil determination condition is to confirm that the area of ​​the detected pupil region is within a set area range, and the second pupil determination condition is to confirm the aggregation of pupil probability values ​​above a threshold value in the detected pupil region with the aid of the pupil probability map, If at least one of the first pupil determination condition and the second pupil determination condition is not met, it is an error setting indicating that the pupil area detection has failed. If both the first pupil determination condition and the second pupil determination condition are met, the detection processing result indicating that the pupil area detection is successful is output.

7. The ophthalmic device according to any one of claims 1 to 6, characterized in that: The ophthalmic device includes a main body with an optical system built in, a driving unit for moving the main body in three dimensions relative to a support unit, and a control unit for controlling various parts of the device. The anterior segment camera is composed of at least two cameras which are arranged at the periphery of the objective lens of the main body and whose lens optical axes are tilted toward the anterior segment of the eye to be examined. The alignment control unit of the control unit that performs control to adjust the relative positional relationship between the eye to be examined and the main body unit includes the pupil detection processing unit and an automatic alignment unit that performs automatic alignment of the pupil. The pupil detection processing unit detects a pupil area using the camera image from the anterior ocular camera and the learned pupil area prediction model set by the learned model setting unit, When the automatic alignment unit automatically aligns the pupil, if the pupil area of ​​the examined eye is successfully detected, two pupil position coordinates are obtained, and a movement instruction is output to the drive unit to make the three-dimensional current coordinates of the pupil calculated based on the two pupil position coordinates converge to the three-dimensional target coordinates of the pupil.

Citation Information

Patent Citations

  • Ophthalmic apparatus

    JP2013248376A

  • Fundus camera and fundus image shooting method

    CN112220447A

  • Eyelid topology morphological feature extraction method based on deep learning

    CN112837805A

  • Ophthalmologic apparatus and control method thereof

    CN114615922A

  • Ophthalmic device, method for controlling ophthalmic device, and program

    WO2021029231A1