Mobile device and method for determining an eye orientation of the eyes of a patient
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- TOOZ TECH GMBH
- Filing Date
- 2024-07-19
- Publication Date
- 2026-06-17
Smart Images
Figure EP2024070610_13022025_PF_FP_ABST
Abstract
Description
[0001] MOBILE DEVICE AND METHOD FOR DETERMINING EYE ALIGNMENT OF A PATIENT'S EYES
[0002] Provided are a computer-implemented method for determining the alignment of a patient's eyes, for use with a mobile device having a display unit and a camera arranged on the same side of the mobile device as the display unit, a computer program, a computer-readable medium, and a mobile device. The disclosure thus lies in the field of ophthalmology and, in particular, in devices for assessing a patient's possible visual impairments.
[0003] Methods and devices are known in the art which can be used by third parties to determine the presence and optionally the severity of strabismus, often also referred to as squinting, in a patient. Such examinations are often referred to as the Hirschberg test or Krimsky test. In conventional procedures, an examining ophthalmologist typically shines a penlight onto the patient centrally at the level of the bridge of the nose. The light source reflects off the cornea of both eyes, where it is visible as a point of light. Since the fovea, the area of sharpest vision, is typically not located centrally on the retina, the reflex point is usually slightly shifted nasally in both eyes due to an angle kappa. In healthy eyes, this decentration is usually the same. If strabismus (squinting) is present in one eye, the reflex point is shifted horizontally and / or vertically.During a medical examination, the severity of any strabismus is assessed by the doctor based on the extent of the kappa angle.
[0004] Devices are also known in the prior art that can facilitate the physician's performance of such an examination. These devices can capture an image of the eyes and subsequently use the images to determine a horizontal and / or vertical displacement of the reflex points. This can be done using dedicated devices provided specifically for this purpose. Alternatively, methods are also known in the prior art in which the physician, with the aid of a mobile device such as a smartphone, captures an image of the eyes and determines any displacement of the reflex points. One such method is described, for example, in US Pat. No. 10,849,492 B2.A rear camera of the mobile device, i.e., a camera located on the side of the mobile device facing away from the display, with an associated flash device, or a front camera with a separate, external light source can be used to capture images of reflex points in a patient's eyes. The reflex points are generated by a flash light, which can be provided by a flash function of the rear camera of the mobile device. The images are recorded by a physician who can operate the mobile device on the side of the mobile device facing the display. A similar method is described in the following publication:
[0005] Pundlik, S., et al. (2019): Development and preliminary evaluation of a smartphone app for measuring eye alignment. Translational vision science & technology, 8 (1), 19-19.
[0006] The object of the present disclosure is to provide a method and a mobile device suitable for enriching the prior art. In particular, the object may be to simplify the implementation of a method for determining the alignment of a patient's eyes.
[0007] This object is achieved by a method, a computer program, a computer-readable storage medium, and a mobile device having the features of the respective independent claim. Advantageous embodiments are specified in the subclaims and in the description. A computer-implemented method for determining an eye alignment of a patient's eyes is provided, for implementation by a mobile device. The mobile device has a display unit and a camera, wherein the camera is arranged on the same side of the mobile device as the display unit. The method comprises issuing instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes such that the display unit and the camera are positioned in a predetermined range relative to the patient's eyes and face the eyes.The method also includes determining a position of the eyes relative to the mobile device, displaying a luminous visualization on the display unit, and capturing at least one image of the eyes using the camera such that a corneal reflection of the luminous visualization displayed on the display unit can be determined in each of the patient's eyes in the captured image. Furthermore, the method includes determining a position of the respective corneal reflection in the eye based on the captured at least one image and determining the eye alignment of both eyes using the position of the respective corneal reflection in the eye.
[0008] Furthermore, a computer program and / or a computer-readable medium is provided containing instructions which, when executed by a mobile device having a display unit and a camera arranged on the side of the display unit, cause the mobile device to perform a method for determining an eye alignment of a patient's eyes. The method comprises issuing instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes such that the display unit and the camera are positioned within a predetermined range relative to the patient's eyes and facing the eyes.The method also includes determining a position of the eyes relative to the mobile device, displaying a luminous visualization on the display unit, and capturing at least one image of the eyes using the camera such that a corneal reflection of the luminous visualization displayed on the display unit can be determined in each of the patient's eyes in the captured image. Furthermore, the method includes determining a position of the respective corneal reflection in the eye based on the captured at least one image and determining the eye alignment of both eyes using the position of the respective corneal reflection in the eye.
[0009] Furthermore, a mobile device for determining an eye alignment of a patient's eyes is provided. The mobile device comprises a display unit and a camera, the latter being arranged on the same side of the mobile device as the display unit. The mobile device is configured to issue instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes such that the display unit and the camera are positioned in a predetermined area relative to the patient's eyes and facing the eyes. Furthermore, the mobile device is configured to determine a position of the eyes relative to the mobile device, display a luminous visualization on the display unit, and capture at least one image of the eyes using the camera such that a corneal reflection of the luminous visualization displayed on the display unit can be determined in each of the patient's eyes in the captured image.In addition, the mobile device is configured to determine a position of the respective corneal reflection in the eye based on the captured at least one image and to determine the eye alignment of both eyes using the position of the respective corneal reflection in the eye.
[0010] Determining the eye alignment can include determining a visual axis of the respective eye. Optionally, determining the eye alignment can include determining a deviation in the eye alignment between the two eyes of the patient. Optionally, determining the eye alignment can include determining a facial line of the respective eye. The facial line can refer to an axis of the eye that runs through the fovea centralis of the eye, through a nodal point of the eye, to an object fixated by the eye. Optionally, determining the eye alignment can include determining a deviation of the facial line from a pupil axis of the eye. The pupil axis can refer to an axis of the eye that runs through the vertex of the cornea and the center of the pupil of the eye.The deviation between the facial line and the pupil axis can correspond to an angle conventionally referred to as the kappa angle. Eye alignment can optionally be specified as an angle and / or in prism diopters.
[0011] A computer-implemented method is a method in which some or all of the method steps can be carried out by a computer, i.e. by an electronic data processing system. Optionally, all of the method steps can be carried out by a computer. The computer can be in the form of a mobile device. A mobile device can be designed as a mobile computer, optionally as a smartphone or as a tablet computer. A display unit can comprise a display or be designed as such. Optionally, the display unit can be designed as a touchscreen and accordingly be configured to enable user input. The mobile device can optionally have one or more processors and / or one or more data memories.Optionally, the mobile device can be designed to establish a communication connection via one or more connection standards, such as GSM, WIFI, Bluetooth, NFC and / or via wired connection types such as USB, Ethernet, etc.
[0012] The fact that the camera is arranged on the same side of the mobile device as the display unit means that the camera is directed towards the patient when the patient uses the mobile device while viewing the display unit. Optionally, the mobile device can have one or more (further) cameras, which can be arranged on other sides of the mobile device. Optionally, the mobile device can have a front-facing camera and a rear-facing camera, whereby the front-facing camera can be arranged on the same side as the display unit and the rear-facing camera can be arranged on a side of the mobile device opposite the display unit. The front-facing camera can be designed and arranged in such a way that the front-facing camera can capture an image of the patient while the patient is looking at the display unit.In other words, the mobile device can enable the patient to take a "selfie" using the front-facing camera, i.e., a photo taken by the patient of themselves while viewing the display unit. The mobile device can be configured such that the front side, on which the display unit and the front-facing camera are located, does not have a separate flash source for providing flash light for taking pictures. However, the rear side can have such a separate flash source, which, however, is unsuitable for taking pictures using the front-facing camera due to the radiation range from the rear side.
[0013] Issuing instructions to the patient may include issuing instructions and / or messages and / or commands via the display unit. For example, an indication may be given as to whether or not the mobile device is positioned in the predetermined area relative to the eyes. Optionally, the instructions may contain information as to the direction in which the mobile device should be moved and / or swiveled in order to reach the predetermined area and / or to improve the positioning and / or orientation of the mobile device. Optionally, issuing the instructions may additionally or alternatively include issuing acoustic and / or haptic signals. This may enable the patient to understand the instructions even when the patient is not looking at the display unit. The patient support may include user guidance, i.e.that the user receives continuous guidance based on the instructions to achieve the desired goal. Optionally, the mobile device can receive inputs from the patient via the display unit, for example, by activating the touchscreen by touching the display unit. This allows the issuing of instructions and / or the execution of the procedure for determining eye alignment to be started, stopped, and / or paused.
[0014] The position of the eyes relative to the mobile device can optionally include an angular range in which the eyes extend relative to the camera or another element of the mobile device and / or relative to a visualization displayed by the display element. Optionally, the position can include a distance of the eyes from the mobile device. Optionally, the mobile device can be positioned at a distance of 20 cm to 70 cm, and optionally, between 30 cm and 50 cm, from the eyes.
[0015] A luminous visualization can represent a light-emitting representation by the display element. Optionally, the luminous visualization can comprise a symbol and / or a pattern or be designed as such. The luminous visualization can emit light in such a way that the luminous visualization generates a corneal reflection that can be captured with the at least one image, which can then be used to determine the respective corneal reflection based on the at least one image.
[0016] The disclosure offers the advantage of enabling a patient to determine eye alignment as part of a self-test using a commercially available mobile device. This offers the particular advantage that the patient requires neither special hardware nor another person to determine eye alignment. Rather, the patient can determine eye alignment themselves, optionally using their own smartphone. Consequently, the hardware requirements are minimal and do not require any personnel, in particular no specially trained personnel, such as an ophthalmologist. Thus, the disclosure offers the advantage that the patient can determine any deviation of the facial line from a pupil axis for each of the two eyes as part of a self-test.Furthermore, the disclosure offers the advantage that by giving instructions to the patient, the patient can be supported in carrying out the project and thus no special skills, knowledge or experience of the patient are required.
[0017] This offers the advantage that, with the patient's assistance, an automated determination of eye alignment can be performed, particularly in aligning the mobile device relative to their eyes. This simplifies the process of determining eye alignment and allows the patient to perform it on demand or routinely with minimal effort.
[0018] Furthermore, the disclosure offers the advantage that such a determination of eye alignment can be implemented in applications of the mobile device. This can optionally be advantageous for applications in which binocular virtual imaging is carried out for the patient or user, such as when using the mobile device for augmented reality applications. Binocular virtual imaging can represent such an application in which different areas of the display unit are viewed by only one of the patient's two eyes, and a three-dimensional visual experience is provided to the patient or user through targeted output of visual content in the different areas of the display unit. Optionally, a planned binocular virtual imaging for the patient can be adapted taking the eye alignment into account in order to improve the visual experience for the patient.This can optionally be advantageous if the patient suffers from heterophoria(s) or heterotopia(s) (strabismus). Stereoscopic vision can be assessed and optionally measured using the kappa angle. Optionally, an assessment of the extent and / or quality of the user's spatial, three-dimensional vision can be given. Optionally, based on the determined eye alignment, a recommendation can be given as to whether or not the patient should view binocular virtual imaging. Optionally, the patient's eye alignment can also be determined using the mobile device in order to provide the patient with binocular virtual imaging on another device and, if necessary, to adapt this based on the determined eye alignment.
[0019] Optionally, the disclosure offers the advantage that a particularly simple detection and / or determination of strabismus can be enabled and thus the selection of a suitable type and / or quality of spectacle lenses and / or other visual aids for the patient can be facilitated and / or improved.
[0020] Furthermore, the disclosure offers the advantage that a rear camera of a mobile device and an associated flash light source are not required for this method, which represents an additional innovation and differentiation from the method of US 10,849,492 B2 and the prior art. The method described in US 10,849,492 B2 requires a flash light source to generate a reflection point. However, since flash light sources in conventional mobile devices are usually only installed with the rear camera, which is facing away from the patient, this makes it very difficult for the patient to carry out the procedure independently and requires an additional light source for carrying out the procedure with the front camera. The method according to the disclosure offers the advantage that it enables the patient to carry out the procedure independently without the need for an additional light source or flash light source.Instead, the corneal reflection can be provided by a luminous visualization displayed on the display unit. Any resulting perspective offset between the camera, eye, and luminous visualization can be relatively corrected by measuring the eye position.
[0021] Determining the position of the respective corneal reflection in the eye can be a
[0022] Determining a relative position of the corneal reflection to a
[0023] Pupil center and / or iris center of each eye. This can offer the advantage of being able to determine or estimate the direction of gaze of the eye. Determining the alignment of both eyes can include determining a deviation of a facial line from a pupil axis for each of the two eyes. In particular, the deviation for both eyes can be compared, and from this, it can be determined whether the patient has strabismus and, if so, how severe the strabismus is.
[0024] The luminous visualization displayed on the display unit can have such a brightness and / or size on the display unit that the luminous visualization can be determined by means of an image analysis of the at least one image by the mobile device. The image analysis can optionally be performed by the mobile device itself, i.e., with one or more processors of the mobile device. Alternatively or additionally, the image can be analyzed by a server remote from the mobile device. For this purpose, the mobile device can transmit the captured image to the server via a network connection and receive the result of the image analysis from the server.
[0025] The method can optionally further comprise displaying a fixation visualization on the display unit prior to and / or during the display of the luminous visualization and the acquisition of the at least one image, in order to direct the patient's gaze to the fixation visualization. This can offer the advantage that the eyes adopt a predetermined viewing direction when the image is acquired, and correspondingly predetermined conditions can be created for determining the position of the respective corneal reflection.
[0026] Determining the position of the eyes relative to the mobile device can include determining, optionally three-dimensionally, an optionally spatial position of the eyes relative to the camera and / or relative to the fixation visualization. This can offer the advantage that a distance and / or positioning of the patient relative to the mobile device during the procedure can be reliably detected, a perspective offset of the light visualization relative to the eyes and / or relative to the camera can be corrected, and any parallax can be taken into account, thereby improving the accuracy of determining the eye alignment.
[0027] Instructions can be issued to the patient via the display unit. Optionally, the instructions can include graphical user interaction. This offers the advantage that no additional hardware is required to issue the instructions. Furthermore, this can offer the advantage that the patient already focuses their gaze on the display unit when viewing the instructions. If the mobile device is correctly aligned relative to the eyes, the illuminated visualization is then also displayed. This can thus promote efficient implementation of the procedure.
[0028] The display unit and the camera are optionally located on the side of the mobile device that faces the patient's eyes when the mobile device is being used. The display unit can be configured as a display and optionally as a touchscreen. Furthermore, the mobile device can be configured to allow the patient to determine the patient's eye alignment as part of a self-test.
[0029] Because the illuminated visualization is displayed on the display unit, the display unit can be used as a light source for generating the corneal reflection. This can offer the advantage that no further separate light source needs to be provided. Furthermore, this can offer the advantage that the illuminated visualization displayed on the display unit can be adjusted in terms of its position on the display unit, its brightness, and its size. Optionally, the illuminated visualization can have a symbol and / or a pattern or be designed as such. Optionally, the symbol and / or pattern can be designed such that their corneal reflection can be reliably recognized in the at least one captured image and, optionally, can be reliably distinguished from any other corneal reflections.Optionally, the symbol may comprise or be formed as one or more of the following geometric shapes: a point, a circle, an ellipse, a polygon, a triangle, a quadrilateral, a square, a rectangle, a trapezoid, a rhombus, and a cushion shape. Optionally, the pattern may comprise or be formed as one or more of the following designs: a grid, a ring structure, a meander, a star, a trademark, a letter, a number, and a symbolic figure. These examples are intended merely to indicate exemplary designs of the illuminated visualization and are not to be interpreted as limiting. The use of other shapes for symbols and patterns is possible.
[0030] Optionally, the luminous visualization can have an optical brightness contrast to the surroundings on the display unit that lies within a predetermined range. Optionally, the luminous visualization can be displayed with a brightness such that the luminous visualization has a brightness contrast relative to the surroundings of the luminous visualization on the display unit of at least 3:1, optionally at least 5:1, optionally at least 10:1, optionally at least 20:1, optionally at least 50:1, optionally at least 100:1, and optionally at least 1000:1.
[0031] Optionally, while the illuminated visualization is being displayed and / or while the at least one image is being captured, the display unit outside the illuminated visualization can be at least partially or completely darkened. This can further increase the brightness contrast and offer the advantage that determining the position of the corneal reflection of the illuminated visualization can be further facilitated. Optionally, the illuminated visualization can be displayed on the display unit in such a way that the illuminated visualization has a pronounced color contrast relative to the surroundings of the illuminated visualization on the display unit. For example, the illuminated visualization can be displayed in one or more colors that are complementary to the colors of any other elements displayed on the display unit. For example, the illuminated visualization can optionally be displayed in white, while surrounding elements on the display unit are displayed in black.For example, the light visualization can optionally be displayed in green or blue, while surrounding elements on the display unit are displayed in red, or vice versa. These examples are intended only to illustrate the use of color contrast and are not intended to be limiting. Other color combinations are possible.
[0032] The features and embodiments mentioned above and explained below are not only to be regarded as disclosed in the respective explicitly mentioned combinations, but are also encompassed by the disclosure content in other technically meaningful combinations and embodiments.
[0033] All disclosures described for the method are also to be considered as disclosed for the computer program and the mobile device and vice versa.
[0034] Further details and advantages of the invention will now be explained in more detail with reference to the following examples and preferred embodiments with reference to the figures.
[0035] They show:
[0036] Fig. 1 shows a schematic representation of a mobile device according to an optional embodiment; Fig. 2 shows a method for determining eye alignment according to an optional embodiment;
[0037] 3A-3F illustrate optional aspects of the method for determining eye alignment according to an optional embodiment;
[0038] In the following figures, identical or similar elements in the various embodiments are designated by identical reference numerals for the sake of simplicity.
[0039] Figure 1 shows a schematic representation of a mobile device 10 for determining the alignment of a patient's eyes according to an optional embodiment. The mobile device 10 comprises a display unit 12 and a camera 14 arranged on the same side of the mobile device 10 as the display unit 12. The display unit 12 and the camera 14 can be arranged on a side of the mobile device 10 that faces the patient's eyes when the patient is using the mobile device 10. This can be the front side 10a of the mobile device.
[0040] The mobile device is configured to perform the following steps:
[0041] (i) issuing instructions to the patient to assist the patient in aligning the mobile device 10 relative to the patient's eyes such that the display unit 12 and the camera 14 are positioned in a predetermined range relative to the patient's eyes and face the eyes.
[0042] (ii) Determining a position of the eyes relative to the mobile device 10.
[0043] (iii) Displaying a light visualization 16 on the display unit 12.
[0044] (iv) capturing at least one image of the eyes by means of the camera 14 such that in the captured image a corneal reflection of the luminous visualization 16 displayed on the display unit 12 can be determined in each of the patient's eyes.
[0045] (v) determining a position of the respective corneal reflection in the eye based on the acquired at least one image and determining the eye alignment of both eyes using the position of the respective corneal reflection in the eye.
[0046] The mobile device 10 can be configured as a smartphone or a tablet computer. The display unit 12 can be configured as a display and optionally as a touchscreen. Optionally, the mobile device 10 can be configured to allow the patient to determine the alignment of their eyes as part of a self-test.
[0047] In particular, the mobile device 10 can be configured to carry out the computer-implemented method described below with reference to Figure 2 according to an optional embodiment.
[0048] Figure 2 schematically shows a computer-implemented method 200 for determining an eye alignment of a patient's eyes for implementation by a mobile device 10 having a display unit 12 and camera 14 arranged on the same side of the mobile device 10 as the display unit 12.
[0049] In a step 202, the method 200 comprises issuing instructions 18 to the patient to assist the patient in aligning the mobile device 10 relative to the patient's eyes such that the display unit 12 and the camera 14 are positioned in a predetermined range relative to the patient's eyes and are facing the eyes. The issuing 202 of the instructions to the patient can optionally be performed via the display unit 12 and can optionally include graphical user interaction. The issuing of instructions can optionally include displaying symbols and / or text content to the patient to instruct the patient to change or maintain a positioning of the mobile device relative to the patient's eyes.For example, such symbols, as shown, can include one or more arrows indicating a direction in which the patient must move the mobile device to achieve the intended positioning. The selection and / or type and / or size and / or color of the displayed symbols can change during the process of aligning the mobile device 10 and adapt to the new positioning of the mobile device during and / or after a movement by the patient. For example, the mobile device can be moved upwards according to the displayed symbols of the instructions 18, as can be indicated by a corresponding upward-pointing arrow. The fact that the other arrows are shown in dashed lines can mean that they are not displayed with the current positioning of the mobile device 10 relative to the eyes.
[0050] In a step 204, the method comprises determining a position of the eyes relative to the mobile device and / or relative to the light visualization. Determining 204 the eye alignment of both eyes may comprise determining a deviation of a facial line from a pupil axis for each of the two eyes. Steps 202 and 204 may optionally be performed at least partially in parallel, i.e., with temporal overlap.
[0051] In a step 206, the method 200 comprises displaying a luminous visualization 16 on the display unit 12. The luminous visualization 16 can have such a brightness and / or size on the display unit 12 that the luminous visualization 16 can be determined by means of image analysis of the at least one image by the mobile device 10.
[0052] In a step 208, the method 200 comprises detecting at least one
[0053] Image of the eyes by means of the camera 14 such that in the captured image a corneal reflection of the luminous visualization 16 can be determined in each of the patient's eyes.
[0054] In a step 210, the method 200 comprises determining a position of the respective corneal reflection in the eye based on the acquired at least one image and determining the eye alignment of both eyes using the position of the respective corneal reflection in the eye. Determining 210 the position of the respective corneal reflection in the eye may comprise determining a relative position of the corneal reflection to a pupil center and / or iris center of the respective eye.
[0055] Optionally, the method 200 can further comprise displaying 212 a fixation visualization 20 on the display unit 12 before and / or during the display 206 of the luminous visualization 16 and the capturing 208 of the at least one image in order to direct the patient's gaze to the fixation visualization 20. The fixation visualization can optionally comprise the display of a stimulus point, which can optionally be displayed in a flickering manner. Optionally, step 212 can further comprise capturing and evaluating one or more images to determine whether the patient is directing their gaze to the fixation visualization or not. If this is not the case, step 212 can optionally be repeated one or more times. The fixation visualization 20 can optionally be displayed centrally on the display unit 12 between the camera 14 and the luminous visualization 16.The position at which the fixation visualization is to be displayed can optionally be determined based on information from the properties of the display unit 12 and / or the position of the camera in the mobile device 10. Optionally, the information to be used to determine the position at which the fixation visualization is to be displayed can be provided in advance, for example, in a provided data set. Determining 204 the position of the eyes relative to the mobile device 10 can include determining a position of the eyes relative to the camera 14 and / or relative to the fixation visualization 20.
[0056] The method 200 may be configured such that the method enables the patient to perform a self-test.
[0057] Further optional aspects are explained below using Figures 3A to 3F.
[0058] Figures 3A and 3B show a mobile device 10 according to an optional embodiment from the patient's perspective when determining an eye alignment using the mobile device 10.
[0059] Figure 3A shows an optional visualization of the captured camera image during the determination of the position of the eyes relative to the mobile device. This can allow the patient to check the detected eye positions in the captured image in order to capture a new image or abort the process if the eyes are incorrectly identified.
[0060] Figure 3B shows an example of the display of a fixation visualization 20 on the display unit 12, the purpose of which is to direct the patient's gaze to the fixation visualization.
[0061] Figure 3C shows, by way of example, the display of a luminous visualization 16 on the display unit 12 during the acquisition of at least one image of the eyes by means of the camera 14. This allows a corneal reflection of the luminous visualization in the acquired image to be determined in each eye. The fixation visualization 20 can be displayed immediately before and / or during the display of the luminous visualization 16 in order to prevent the patient from averting their gaze from the fixation visualization. As shown by the dashed lines, the camera 14, the fixation visualization 20, and the luminous visualization 16 can have a predetermined arrangement relative to one another, which can be used to determine the position of the respective corneal reflection.
[0062] The luminous visualization can be displayed briefly immediately after the fixation visualization is displayed, for example, 100 ms to 500 ms afterward, and then one or more images can be captured immediately by the camera. Depending on the quality, a video frame can also be used for this purpose; however, it is assumed that a photo can provide higher image quality than a video frame. By displaying the luminous visualization 16 very quickly, it is achieved that the eye's fixation point is still aligned by the fixation visualization and has not yet been directed to the pattern now visible on the display unit.
[0063] Figure 3D shows a schematic side view of the relative positioning of the mobile device 10 to the eyes, which may be suitable for determining eye alignment. The lines visualize the direction vectors from the eye to the light visualization 16, the fixation visualization 209, and the camera 14. These can be incorporated into the determination of the respective eye alignment to account for any paralex error.
[0064] Figure 3E shows, by way of example, an image of an eye 24 of the patient 22, in which a corneal reflection 26 of the illuminated visualization 16 is visible. This can optionally be derived from the captured image. It can be seen that the illuminated visualization 16 is not located on the pupil center and / or iris center indicated by the crossed lines, but is offset horizontally and vertically from the pupil center and / or iris center.
[0065] Based on this, a vertical and / or horizontal displacement of a visual line of the eye from a pupil axis of the eye can be determined. This can optionally be calculated by the mobile device 10. The schematically illustrated distances and direction vectors, which are plotted in Figure 3D, are explained in more detail schematically and by way of example in Figure 3F. Line 300 indicates the connecting vector between a center point or other designated point of the light visualization and the eye 24, for example the pupil center point and / or iris center point, and line 302 indicates the connecting vector between the camera 14 and the eye 24. Arrow 304 indicates the distance of the eye 24 from the front side 10a of the mobile device 10 in the z-direction, and arrow 306 indicates the distance of the eye from the camera 14 in the y-direction. Optionally, a further distance in the x-direction between the camera 14 and the eye 24 can be determined.The eyes can be positioned centrally relative to the camera, meaning that the distance between the bridge of the nose and the camera in the x-direction is approximately zero. This can result in the x-distance of one of the eyes being approximately between 3.15 cm and -3.15 cm.
[0066] Optionally, the patient can be instructed to position and align the mobile device at a predetermined position such that the distance of the eye 24 in the y-direction from the camera 14 and from the center of the illuminated visualization 16 are approximately equal. For this purpose, the y-distance of the eye from the camera can optionally be approximately 2 cm. The predetermined position of the eye can optionally be approximately 40 cm from the front of the mobile device.
[0067] To determine the fixation direction of eye 24, i.e., the line of sight, the position of the respective eye relative to the mobile device is relevant. Optionally, a three-dimensional position of the eye relative to the mobile device can be calculated as follows:
[0068] Variant A) Based on a depth sensor built into the mobile device, a depth map is created and, if necessary, adjusted in perspective to the image section of the camera 14. With the help of camera-based face recognition (algorithmic solution), the position of the individual eye can be detected in a two-dimensional projection in the captured image. Based on the detected two-dimensional position in the captured image, the distance of the eye in the z-direction can be derived from the depth map (=> z-component of the 3D position). Based on an intrinsic calibration of the camera and the two-dimensional position of the individual eye in the camera image, an x- and y-component of the three-dimensional position of the individual eye can be calculated.
[0069] Variant B), described below, is similar to variant A in many aspects, but alternatively, the x- and y-position components are also taken from the depth map. This allows for consideration of whether a depth map has been perspectively distorted by adjusting it to the image section of the front camera. This can be corrected if necessary.
[0070] Furthermore, variant C) is described below as an alternative without a depth sensor. Here, the z-distance is calculated from the captured camera image. For calibration, a known object size may be necessary, which must be located at least once in the camera image section at a similar distance to the eye (e.g., a credit card can be held in front of the eye or against the forehead). This can be used to determine the current z-distance from the detected size of the object. Using the detected distance, the pupil size and / or iris size of the individual eye in the camera image can be determined. This can then be used as a reference value for determining the distance in the z-direction. The x and y components of the position can be determined in the same way as variant A.
[0071] Furthermore, variant D) is described below as an alternative without a depth sensor and without a reference object. Here, the z-distance is calculated from the captured camera image, and an iris size and / or pupil size is used as a reference value to determine the distance in the z-direction. The determination of an x and y component of the position can be carried out in the same way as variant A. The position of the eyes can optionally be determined regularly at defined intervals. Optionally, the position is determined immediately before the light visualization is displayed. Optionally, the eye position can also be determined while instructions are given to the patient to assist in aligning the mobile device, in order to always obtain precise and updated knowledge of any deviation from the positioning of the mobile device within the predetermined range relative to the eyes.
[0072] As soon as the eyes are in the predetermined three-dimensional position relative to the mobile device, or as soon as the mobile device is in the predetermined area relative to the eyes - for this purpose, threshold values with regard to the three-dimensional position (x, y, z) can optionally be specified - the fixation direction of both eyes can also be controlled.
[0073] A perspective offset between the illuminated visualization on the display unit and the camera's position relative to the eye can be compensated for by having the patient fixate their gaze centrally between the camera and the illuminated visualization. This can optionally be achieved by guiding the gaze direction with a fixation visualization, such as a flashing visual stimulus. The stimulus can optionally flash rapidly for a defined period of time (e.g., 2 seconds). The procedure can then be continued automatically.
[0074] The position of the flashing stimulus is optionally calculated from the position of the camera, the position of the light visualization on the display unit, and the position of the eye to be measured. These optionally form an isosceles triangle (see Figure 3F), with the flashing stimulus located centrally between the camera and the pattern, at the level of both eyes. The corneal reflection of the light visualization of the respective eye can be determined in relation to the pupil center and / or iris center, for example using algorithmic image processing, and a horizontal and / or vertical deviation can be determined between the corneal reflection and the pupil center and / or iris center. The deviation can optionally be specified in mm and / or in prism diopters. For this purpose, the kappa angle, which indicates the difference between the optical axis and the line of sight, can now be calculated.
[0075] The corneal reflection is usually slightly nasally shifted horizontally in both eyes due to the kappa angle. This shift can be taken into account when determining the horizontal / vertical shift of the facial line, optionally assuming that at least one eye is free of strabismus.
[0076] The facial line can optionally be perspectively corrected by a known perspective offset from the known three-dimensional relation of the respective eye, the respective light visualization and the respective camera.
[0077] List of reference symbols
[0078] 10 mobile devices
[0079] 10a Front of the mobile device
[0080] 12 Display unit
[0081] 14 Camera
[0082] 16 Light visualization
[0083] 18 Instruction
[0084] 20 Fixation visualization
[0085] 22 patients
[0086] 24 Patient's eye
[0087] 26 Corneal reflection
[0088] 200 Methods for Determining Eye Alignment
[0089] 202-212 Procedural steps
[0090] 300 Connection vector between the center of the light visualization and the eye
[0091] 302 Connection vector between camera and eye
[0092] 304 z-distance between front of mobile device and eye
[0093] 306 y-distance between camera and eye
Claims
Patent claims 1. A computer-implemented method (200) for determining an eye alignment of the eyes (24) of a patient (22) for implementation by a mobile device (10) having a display unit (12) and a camera (14) located on the same side of the mobile device (10) as the display unit (12), characterized in that the method (200) comprises: - issuing (202) instructions (18) to the patient (22) to assist the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined range relative to the eyes (24) of the patient (22) and face the eyes (24); - determining (204) a position of the eyes (24) relative to the mobile device (10); - displaying (206) a light visualization (16) on the display unit (12); - capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the luminous visualization (16) displayed on the display unit (12) can be determined in each of the eyes (24) of the patient (22); and - Determining (210) a position of the respective corneal reflection (26) in the eye (24) based on the captured at least one image and determining the eye alignment of both eyes (24) using the position of the respective corneal reflection (26) in the eye (24).
2. The method (200) according to claim 1, wherein determining the position of the respective corneal reflection (26) in the eye (24) comprises determining a relative position of the corneal reflection (26) to a pupil center and / or iris center of the respective eye (24).
3. The method (200) of claim 1 or 2, wherein determining the eye alignment of both eyes (24) comprises determining a deviation of a line of sight from a pupil axis for each of the two eyes (24).
4. Method (200) according to one of the preceding claims, wherein the luminous visualization (16) has such a brightness and / or size on the display unit (12) that the luminous visualization (16) can be determined by means of image analysis of the at least one image by the mobile device (10).
5. Method (200) according to one of the preceding claims, further comprising: - Displaying (212) a fixation visualization (20) on the display unit (12) before and / or during the display (206) of the luminous visualization (16) and the capture (208) of the at least one image in order to direct the patient's (22) gaze to the fixation visualization (20).
6. The method (200) according to any one of the preceding claims, wherein determining (202) the position of the eyes (24) relative to the mobile device (10) comprises determining a position of the eyes (24) relative to the camera (14) and / or relative to the fixation visualization (20).
7. The method (200) according to any one of the preceding claims, wherein the output (202) of the instructions (18) to the patient (22) is carried out by means of the display unit (12) and optionally comprises a graphical user interaction.
8. The method (200) according to any one of the preceding claims, wherein the method (200) is designed such that the method (200) enables the patient (24) to perform a self-test.
9. Computer program and / or computer-readable medium, characterized in that the computer program and / or the computer-readable medium contain instructions which, when executed by a mobile device (10) having a display unit (12) and a camera (14) arranged on the side of the display unit (12), cause the mobile device (10) to perform a method for determining an eye alignment (26) of the eyes (24) of a patient (22), the method (200) comprising: - issuing (202) instructions (18) to the patient (22) to assist the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined range relative to the eyes (24) of the patient (22) and face the eyes (22); - determining (204) a position of the eyes (24) relative to the mobile device (10); - displaying (206) a light visualization (16) on the display unit (12); - capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the luminous visualization (16) displayed on the display unit (12) can be determined in each of the eyes (24) of the patient (22); and - determining (210) a position of the respective corneal reflection (26) in the eye (24) based on the captured at least one image and determining the eye alignment of both eyes (24) using the position of the respective corneal reflection (26) in the eye (24).
10. A mobile device (10) for determining an eye alignment of the eyes (24) of a patient (22), the mobile device (10) comprising: - a display unit (12); and - a camera (14) arranged on the same side of the mobile device (10) as the display unit (12); characterized in that the mobile device (10) is configured to perform the following steps: - issuing (202) instructions (18) to the patient (22) to assist the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined range relative to the eyes (24) of the patient (22) and face the eyes (24); - determining (204) a position of the eyes (24) relative to the mobile device (10); - displaying (206) a light visualization (16) on the display unit (12); - capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the luminous visualization (16) displayed on the display unit (12) can be determined in each of the eyes (24) of the patient (2) in the captured image; and - Determining (210) a position of the respective corneal reflection (26) in the eye (24) based on the captured at least one image and determining the eye alignment of both eyes (24) using the position of the respective corneal reflection (26) in the eye.
11. Mobile device (10) according to claim 10, wherein the mobile device (10) is designed as a smartphone or as a tablet computer.
12. Mobile device (10) according to claim 10 or 11, wherein the display unit (12) and the camera (14) are arranged on a side of the mobile device (10) which, when the mobile device (10) is used by the patient (22), faces the eyes (24) of the patient (22).
13. Mobile device (10) according to one of claims 10 to 12, wherein the display unit (12) is designed as a display and optionally as a touchscreen.
14. Mobile device (10) according to one of claims 10 to 13, wherein the Mobile device (10) is configured to enable the determination of the eye alignment of the eyes (24) of the patient (22) as part of a self-test by the patient (22).