Mobile device and method for determining eye orientation of eye of patient

Abstract

The invention relates to a computer-implemented method (200) for determining an eye orientation of an eye (24) of a patient (22), performed by a mobile device (10) having a display unit (12) and a camera (14) arranged on the same side of the mobile device (10) as the display unit (12). The method (200) includes outputting (202) instructions (18) to the patient (22) for assisting the patient (22) when orienting the mobile device (10) relative to the eye (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined area relative to the eye (24) of the patient (22) and face the eye (24). The method further comprises determining (204) a position of the eye (24) relative to the mobile device (10); displaying (206) the light visualization (16) on the display unit (12); and capturing (208) at least one image of the eyes (24) by means of the camera (14), such that a corneal reflection (26) of the light visualization (16) displayed on the display unit (12) in each eye (24) of the patient (22) can be determined in the captured image. Furthermore, the method comprises determining (210) a position of a respective corneal reflection (26) in the eye (24) based on the captured at least one image, and determining an eye orientation of the two eyes (24) using the position of the respective corneal reflection (26) in the eye (24).

Description

Technical Field

[0001] This invention relates to a computer-implemented method for determining the eye alignment of a patient's eye, the method being executed by a mobile device having a display unit and a camera arranged on the same side of the mobile device as the display unit; the invention also relates to a computer program, a computer-readable medium, and a mobile device. Therefore, this disclosure pertains to the field of ophthalmology, and more specifically to the field of devices for assessing possible refractive errors in patients. Background Technology

[0002] Existing technology has disclosed methods and apparatus that can be used by third parties to determine whether a patient has strabismus (often referred to as misogyny) and optionally to determine the degree of strabismus. This examination is often called the Hirschberg test or the Krimsky test. In the routine method, the ophthalmologist performing the examination typically shines a pen-shaped flashlight centered at the bridge of the patient's nose. The light source reflects off the corneas of both eyes and appears as a point of light in each case. Because the fovea, representing the area of ​​clearest vision, is not usually located at the center of the retina, the reflected point on both eyes is typically displaced slightly nasally by a kappa angle. For healthy eyes, this displacement from the center is usually the same. If strabismus (misogyny) is present, the reflected point is displaced horizontally and / or vertically. In the context of the medical examination, the physician estimates the degree of possible strabismus based on the appearance of the kappa angle.

[0003] Prior art also discloses devices that facilitate such examinations by physicians, wherein these devices can capture images of the eye and subsequently determine the horizontal and / or vertical displacement of the reflective point based on these images. This can be implemented using dedicated equipment specifically provided for this purpose. Alternatively, prior art also discloses methods by which physicians capture images of the eye and determine any displacement of the reflective point by means of a mobile device, such as a smartphone. For example, such a method is described in US 10,849,492 B2. In this context, images of the reflective point in the patient's eye can be captured using a rear-side camera of the mobile device (i.e., a camera positioned on the side of the mobile device opposite to the display) and an associated flash unit, or using a front-side camera with a separate external light source. In this case, the reflective point is generated by a flash, which can be provided by the flash function of the rear-side camera of the mobile device. The images are recorded by a physician who operates the mobile device on the side containing the mobile device's display. Similar methods are described in the following publications:

[0004] Pundlik, S. et al. (2019): Development and preliminary evaluation of asmartphone app for measuring eye alignment. Translational vision science & technology, 8 (1), 19-19. Summary of the Invention

[0005] In this context, the problem addressed by this disclosure is to provide methods and mobile devices suitable for enhancing existing technologies. In particular, the problem addressed may lie in making it easier to perform methods for determining eye alignment for a patient's eyes.

[0006] This problem is solved by methods, computer programs, computer-readable storage media, and mobile devices having the features of the respective independent claims. Advantageous configurations are described in detail in the dependent claims and the specification.

[0007] This invention provides a computer-implemented method for determining eye alignment of a patient's eyes, the method being performed by a mobile device. In this case, the mobile device includes a display unit and a camera, wherein the camera is arranged on the same side of the mobile device as the display unit. The method includes: outputting instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes, such that the display unit and the camera are positioned relative to the patient's eyes in a predetermined area and facing the eyes. Furthermore, the method includes: determining the position of the eyes relative to the mobile device; displaying a light visualization on the display unit; and capturing at least one image of the eyes by means of the camera, such that the corneal reflections in each eye of the patient can be determined from the captured images of the light visualization displayed on the display unit. Further, the method includes: determining the position of a corresponding corneal reflection in the eye based on at least one captured image, and using the position of the corresponding corneal reflection in the eye to determine the eye alignment of both eyes.

[0008] The present invention also provides a computer program and / or a computer-readable medium comprising instructions that, when executed by a mobile device having a display unit and a camera disposed on the side of the display unit, cause the mobile device to perform a method for determining eye alignment of a patient's eyes. In this case, the method includes: outputting instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes, such that the display unit and the camera are positioned relative to the patient's eyes in a predetermined area and facing the eyes. Furthermore, the method includes: determining the position of the eyes relative to the mobile device; displaying a light visualization on the display unit; and capturing at least one image of the eyes by means of the camera, such that corneal reflections in each eye of the patient can be determined from the captured images of the light visualization displayed on the display unit. Further, the method includes: determining the position of a corresponding corneal reflection in the eye based on at least one captured image, and using the position of the corresponding corneal reflection in the eye to determine eye alignment of both eyes.

[0009] Furthermore, the present invention provides a mobile device for determining eye alignment of a patient's eyes. The mobile device includes a display unit and a camera, wherein the camera is arranged on the same side of the mobile device as the display unit. The mobile device is configured to output instructions to the patient to assist the patient in aligning the mobile device relative to the patient's eyes, such that the display unit and the camera are positioned relative to the patient's eyes in a predetermined area and facing the eyes. Furthermore, the mobile device is configured to: determine the position of the eyes relative to the mobile device; display a light visualization on the display unit; and capture at least one image of the eyes by means of the camera, such that the corneal reflections in each eye of the patient can be determined from the captured images of the light visualization displayed on the display unit. Furthermore, the mobile device is configured to: determine the position of the corresponding corneal reflection in the eye based on at least one captured image, and use the position of the corresponding corneal reflection in the eye to determine eye alignment of both eyes.

[0010] In this context, determining eye alignment may include determining the visual axis of the respective eye. Optionally, determining eye alignment may include determining the deviation of eye alignment between the patient's two eyes. Optionally, determining eye alignment may include determining the line of sight of the respective eye. In this context, the line of sight may represent the axis of the eye's extension through the fovea of ​​the eye, through the nodal point of the eye, and reaching the object the eye is looking at. Optionally, determining eye alignment may include determining the deviation of the line of sight from the pupillary axis of the eye. In this context, the pupillary axis may represent the axis of the eye's extension through the apex of the cornea and the center of the pupil of the eye. In this context, the deviation between the line of sight and the pupillary axis may correspond to an angle commonly referred to as the Kappa angle. Eye alignment may optionally be specified in terms of angle and / or prism diopter.

[0011] In this context, the computer-implemented method is one in which some or all of the method steps can be performed by a computer (i.e., by an electronic data processing device). Optionally, all method steps can be performed by a computer. In this context, the computer can take the form of a mobile device. Here, the mobile device can be implemented as a mobile computer, optionally as a smartphone or tablet computer. The display unit can include or be implemented as a display. Optionally, the display unit can be implemented as a touchscreen and is therefore designed to implement user input. The mobile device may optionally include one or more processors and / or one or more data storage devices. Optionally, the mobile device can be designed to establish a communication link via one or more connectivity standards (e.g., via GSM, WIFI, Bluetooth, NFC, and / or via wired connection types (e.g., USB, Ethernet, etc.)).

[0012] The camera being positioned on the same side of the mobile device as the display unit means that the camera faces the patient when the patient is using the mobile device to view the display unit. Optionally, the mobile device may include one or more (additional) cameras that can be positioned on other sides of the mobile device. Optionally, the mobile device may have a front-side camera and a rear-side camera, wherein the front-side camera may be positioned on the same side as the display unit, and the rear-side camera may be positioned on the opposite side of the mobile device from the display unit. In this case, the front-side camera can be implemented and positioned in a manner that allows the front-side camera to capture an image of the patient while the patient is viewing the display unit. In other words, the mobile device may allow the patient to record a "selfie" (i.e., a self-recording taken by the patient while viewing the display unit) using the front-side camera. In this case, the mobile device may be configured such that the front side of the mobile device, where the display unit and the front-side camera are formed, does not have a separate flash source for providing a flash for recording the image. In contrast, the rear side may include such a separate flash source; however, since the light emission area is from the rear side, such a separate flash source is not suitable for recording images using the front-side camera.

[0013] Providing instructions to the patient may include outputting multiple pieces of information and / or messages and / or commands via a display unit. For example, one piece of information may be output regarding whether the mobile device is positioned relative to the eyes in a predetermined area. Optionally, the multiple pieces of information may include information about the direction in which the mobile device should be moved and / or tilted to reach the predetermined area and / or improve the positioning and / or orientation of the mobile device. In addition, or alternatively, outputting instructions may optionally include outputting acoustic and / or tactile signals. This allows the patient to receive multiple pieces of information even when not looking at the display unit. In this context, providing assistance to the patient may include user guidance, i.e., the user continuously receiving guidance based on instructions to reach the desired goal. Optionally, the mobile device may receive patient input via a display unit (e.g., by actuating a touchscreen due to touching the display unit). As a result, the output of instructions and / or the execution of methods for determining eye alignment may optionally begin and / or end and / or pause.

[0014] The eye's position relative to the mobile device may optionally include an angular range extending from the eye relative to the camera or any other element of the mobile device and / or relative to the visualization displayed by the display element. Optionally, this position may include the distance of the eye from the mobile device. Optionally, the mobile device may be positioned at a distance of 20 cm to 70 cm in front of the eye, and optionally at a distance of 30 cm to 50 cm.

[0015] In this context, the optical visualization can represent the emission reproduction of the display element. Optionally, the optical visualization may include, or be implemented as, symbols and / or patterns. In this case, the optical visualization can emit light in such a way that it produces a corneal reflection, which can be captured by at least one image and can subsequently be used to determine the corresponding corneal reflection based on at least one image.

[0016] The advantage of this disclosure is that it enables patients to determine eye alignment within a self-test range using commercially available mobile devices. Specifically, this offers the advantage that patients do not require special hardware or additional personnel to determine eye alignment. Instead, patients can perform eye alignment determination themselves, optionally using their own smartphones in the process. Therefore, the required hardware is minimal, and no staff, particularly not specially trained personnel such as ophthalmologists, are needed. Thus, the advantage of this disclosure is that, within a self-test range, patients can determine the possible deviation between the visual axis and the pupillary axis for each of their two eyes.

[0017] Furthermore, the advantage provided by this disclosure is that by issuing instructions to the patient, the procedure can be assisted, and therefore it is not necessary to assume that the patient possesses any special skills, knowledge, or experience.

[0018] Therefore, this offers the following advantages: eye alignment can be automatically determined with the patient's assistance, particularly regarding the alignment of the mobile device relative to the patient's eyes. This simplifies eye alignment determination and allows it to be performed periodically or as needed with minimal effort from the patient.

[0019] Furthermore, the advantage provided by this disclosure is that the determination of this eye alignment can be implemented within a mobile device application. This may be advantageous for applications that feature binocular virtual imaging for patients or users (e.g., when the mobile device is used for augmented reality applications). In this case, binocular virtual imaging can represent an application in which, in each case, only one eye of the patient views different areas of a display unit, and the patient or user experiences a three-dimensional visual experience due to the targeted output of visual content in different areas of the display unit. Optionally, in this case, eye alignment can be considered to adapt the binocular virtual imaging designed for the patient in order to improve the patient's visual experience. This may be advantageous, optionally, in cases where the patient has (multiple) latent strabismus or (multiple) manifest strabismus (crossed eyes). Kappa angles can be used to evaluate and optionally measure stereoscopic vision. Optionally, estimates of the performance and / or quality of the user's spatial three-dimensional vision can be provided. Optionally, the determined eye alignment can be used to output a recommendation on whether to recommend the patient to view the binocular virtual imaging. Alternatively, the mobile device can also be used to determine the patient’s eye alignment in order to provide and optionally use the determined eye alignment on different devices to adapt binocular virtual imaging.

[0020] Optionally, the advantage provided by this disclosure is that it can allow for particularly simple identification and / or determination of strabismus, and thereby facilitate and / or improve the selection of appropriate type and / or performance of spectacle lenses and / or other visual aids for patients.

[0021] Furthermore, the advantage provided by this disclosure is that this method does not require a rear-side camera and associated flash source in the mobile device, representing an additional innovation and distinction from the methods described in US 10,849,492 B2 and prior art. The method described in US 10,849,492 B2 requires a flash source to generate the reflection point. However, independent patient implementation becomes more difficult because the flash source in conventional mobile devices is typically only mounted with the rear-side camera, which faces away from the patient, while implementation with a front-side camera requires an additional light source. The advantage provided by the method according to this disclosure is that it allows the method to be implemented independently by the patient without requiring an additional light source or flash source. Instead, corneal reflection can be provided through an optical visualization displayed on the display unit. The perspective shift that may result between the camera, eye, and optical visualization can be relatively corrected by measuring the eye position.

[0022] Determining the position of the corresponding corneal reflex in the eye can include determining the relative position of the corneal reflex with respect to the pupillary center and / or iris center of the corresponding eye. This can provide the advantage of determining or estimating the viewing direction of the eye. Determining eye alignment in both eyes can include determining the deviation of the line of sight from the pupillary axis of each eye. In particular, the deviations of the two eyes can be compared with each other and used to determine whether the patient presents with strabismus and optionally the severity of the strabismus.

[0023] The light visualization displayed on the display unit, including its brightness and / or size, can be determined by performing an image evaluation on at least one image using a mobile device. In this case, the image evaluation can optionally be performed by the mobile device itself (i.e., using one or more processors of the mobile device). Alternatively or otherwise, the image evaluation can be implemented on a server located remotely from the mobile device. For this purpose, the mobile device can send the captured image to the server via a network connection and receive the results of the image evaluation from the server.

[0024] The method may optionally further include displaying a gaze visualization on the display unit before and / or simultaneously with displaying the light visualization and capturing the at least one image, so as to guide the patient's gaze toward the gaze visualization. This can provide the advantage that when the image is captured, the eye can adopt a predetermined viewing direction, and thus predetermined conditions can be correspondingly created for determining the position of the corresponding corneal reflex.

[0025] Determining the position of the eye relative to the mobile device can include determining, optionally in three dimensions, the position of the eye relative to the camera and / or relative to the gaze visualization, and optionally in spatial position. This can provide the following advantages: the distance and / or positioning of the patient relative to the mobile device can be reliably detected during the method, perspective shifts in the optical visualization relative to the eye and / or relative to the camera can be corrected, any potential parallax can be taken into account, and thereby the accuracy of determining eye alignment can be improved.

[0026] Instructions can be output to the patient via a display unit. Optionally, outputting these instructions may include a graphical user interaction. This offers the advantage that no additional hardware is required to output the instructions. Furthermore, this provides the advantage that the patient has already directed their gaze to the display unit for the purpose of observing the instructions, and the display unit is then used to display light visualizations with the mobile device correctly aligned with the eye. Therefore, this can facilitate the efficient execution of the method.

[0027] In this configuration, the display unit and camera are optionally positioned on the side of the mobile device facing the patient's eyes when the patient uses the device. The display unit can be implemented as a monitor and optionally as a touchscreen. Furthermore, the mobile device can be configured to allow determination of the patient's eye alignment within a range of patient self-testing.

[0028] Because the light visualization is displayed on the display unit, the display unit can be used as a light source to generate corneal reflections. This offers the advantage that no additional separate light source is required. Furthermore, it provides the advantage that the light visualization displayed on the display unit is adaptable in terms of its position, brightness, and size on the display unit.

[0029] Optionally, the light visualization may include, or be implemented as, symbols and / or patterns. Optionally, the symbols and / or patterns may be configured such that their corneal reflections are reliably identifiable in at least one captured image and optionally reliably distinguishable from any other corneal reflections. Optionally, symbols may include or be implemented as one or more of the following geometries: points, circles, ellipses, polygons, triangles, quadrilaterals, squares, rectangles, trapezoids, rhombuses, and pincushion shapes. Optionally, patterns may include or be implemented as one or more of the following designs: grids, ring structures, zigzag lines, stars, logos, letters, numbers, and symbolic graphics. These examples are intended only to indicate exemplary configurations of light visualization and should not be construed as limiting. Other forms of symbols and patterns may be used.

[0030] Optionally, the light visualization can have an optical brightness contrast within a predetermined range relative to the surrounding environment on the display unit. Optionally, the brightness display of the light visualization can make the brightness contrast of the light visualization relative to the surrounding environment visualized by the illumination means on the display unit at least 3:1, optionally at least 5:1, optionally at least 10:1, optionally at least 20:1, optionally at least 50:1, optionally at least 100:1, and optionally at least 1000:1.

[0031] Optionally, when displaying light visualization and / or capturing at least one image, the display unit may be at least partially or completely darkened in areas outside the light visualization. This can result in a further increase in brightness contrast and provides the advantage of further simplifying the determination of the location of the corneal reflection in the light visualization.

[0032] Optionally, the light visualization can be displayed on the display unit in such a way that the light visualization has significant color contrast relative to its surrounding environment on the display unit. Therefore, the light visualization can be presented in one or more colors that are complementary to the color of any other element displayed on the display unit. Thus, when the surrounding elements on the display unit are displayed in black, the light visualization can optionally be displayed in white. Thus, when the surrounding elements on the display unit are displayed in red, the light visualization can optionally be displayed in green or blue, and vice versa. These examples are intended only to illustrate the use of color contrast and should not be construed as limiting. Other color combinations are also possible.

[0033] In this context, the features and embodiments mentioned above and explained below should not only be considered as disclosed in the corresponding expressly mentioned combinations, but should also be covered by this disclosure in other technically advantageous combinations and embodiments.

[0034] All disclosures describing this method should also be considered as disclosures for computer programs and mobile devices, and vice versa. Attached Figure Description

[0035] Further details and advantages of the invention will now be explained in more detail with reference to the accompanying drawings, based on the following examples and preferred embodiments.

[0036] In the attached diagram:

[0037] Figure 1 A schematic diagram of a mobile device according to an optional embodiment is shown;

[0038] Figure 2 A method for determining eye alignment according to an optional embodiment is shown;

[0039] Figures 3A to 3FAn optional aspect of a method for determining eye alignment, according to an alternative embodiment, is shown.

[0040] For simplicity, the same or similar elements in the various embodiments are represented by the same reference numerals in the following figures. Detailed Implementation

[0041] Figure 1 A schematic diagram of a mobile device 10 for determining eye alignment of a patient's eye, according to an alternative embodiment, is shown. The mobile device 10 includes a display unit 12 and a camera 14, which is arranged on the same side of the mobile device 10 as the display unit 12. In this case, the display unit 12 and camera 14 can be arranged on the side of the mobile device 10 facing the patient's eyes when the patient uses the mobile device 10. This could be the front side 10a of the mobile device.

[0042] The mobile device is configured to perform the following steps:

[0043] (i) Output instructions to the patient to assist the patient in aligning the mobile device 10 relative to the patient's eyes, such that the display unit 12 and the camera 14 are positioned in a predetermined area relative to the patient's eyes and facing the eyes.

[0044] (ii) Determine the position of the eye relative to the mobile device 10.

[0045] (iii) Display light visualization 16 on display unit 12.

[0046] (iv) At least one image of the eye is captured by means of camera 14, such that the corneal reflection of the light visualization 16 displayed on display unit 12 in each eye of the patient can be determined in the captured image.

[0047] (v) Determine the position of the corresponding corneal reflection in the eye based on at least one captured image, and use the position of the corresponding corneal reflection in the eye to determine eye alignment of both eyes.

[0048] In this configuration, the mobile device 10 can be implemented as a smartphone or tablet computer. The display unit 12 can be implemented as a monitor, and optionally as a touchscreen. Optionally, the mobile device 10 can be configured to allow determination of the patient's eye alignment within the range of patient self-testing.

[0049] In particular, according to an optional embodiment, the mobile device 10 can be configured to perform the following reference. Figure 2 The described computer-implemented method.

[0050] Figure 2A computer-implemented method 200 for determining the eye alignment of a patient is schematically shown. This method is performed by a mobile device 10 having a display unit 12 and a camera 14, which is arranged on the same side of the mobile device 10 where the display unit 12 is also arranged.

[0051] In step 202, method 200 includes outputting instructions 18 to the patient to assist the patient in aligning the mobile device 10 relative to the patient's eyes, such that the display unit 12 and camera 14 are positioned relative to the patient's eyes in a predetermined area and facing the eyes during the process. Outputting instructions 202 to the patient may optionally be implemented by means of the display unit 12 and may optionally include graphical user interaction. The output instructions may optionally include displaying symbols and / or text content to the patient to instruct the patient to change or maintain the positioning of the mobile device relative to the patient's eyes. For example, as shown, such symbols may include one or more arrows indicating the direction in which the patient should move the mobile device to achieve the desired positioning. The selection and / or type and / or size and / or color of the displayed symbols may change during the alignment process of the mobile device 10 and may be adapted to the new positioning of the mobile device during and / or after the patient makes a movement. For example, it may be necessary to move the mobile device upwards according to the symbols displayed in instruction 18 (such as those indicated by corresponding upward arrows). The fact that other arrows are indicated by dashed lines may mean that these arrows are not displayed in the current positioning of the mobile device 10 relative to the eyes.

[0052] In step 204, the method includes determining the position of the eyes relative to the mobile device and / or relative to light visualization. Determining eye alignment for both eyes in step 204 may include determining the deviation of the line of sight from the pupil axis for each eye. Steps 202 and 204 may optionally be performed at least partially in parallel, i.e., in cases of temporal overlap.

[0053] In step 206, method 200 includes displaying a light visualization 16 on the display unit 12. The brightness and / or size of the light visualization 16 on the display unit 12 are such that the light visualization 16 can be determined by performing an image evaluation on at least one image using the mobile device 10.

[0054] In step 208, method 200 includes capturing at least one image of the eye by means of camera 14, such that the corneal reflection of light visualization 16 in each of the patient's eyes can be determined in the captured image.

[0055] In step 210, method 200 includes determining the position of a corresponding corneal reflection in the eye based on at least one captured image, and using the position of the corresponding corneal reflection in the eye to determine eye alignment of the two eyes. Determining the position of the corresponding corneal reflection in the eye may include determining the relative position of the corneal reflection with respect to the pupil center and / or iris center of the corresponding eye.

[0056] Optionally, method 200 may further include displaying gaze visualization 20 on display unit 12 before and / or simultaneously with displaying 206 light visualization 16 and capturing 208 the at least one image, in order to guide the patient's gaze to gaze visualization 20. Gaze visualization may optionally include the display of a stimulus point, which may optionally be displayed in a flashing manner. Optionally, step 212 may further include capturing and evaluating one or more images to determine whether the patient is guiding their gaze to the gaze visualization. If not, step 212 may optionally be repeated once or multiple times. Gaze visualization 20 may optionally be centered on display unit 12 between camera 14 and light visualization 16. The location where gaze visualization should be displayed may optionally be determined based on information about the characteristics of display unit 12 and / or the position of the camera in mobile device 10. Optionally, the information for determining the location where gaze visualization should be displayed may be provided in advance, for example, in the provided data record.

[0057] Determining the position of the 204 eyes relative to the mobile device 10 may include determining the position of the eyes relative to the camera 14 and / or relative to the gaze visualization 20.

[0058] Method 200 can be configured in a way that allows patients to perform self-testing.

[0059] The following text is based on Figures 3A to 3F Other optional aspects are explained.

[0060] Figure 3A and Figure 3B The mobile device 10 is shown from the patient's perspective when eye alignment is determined by means of the mobile device 10, according to an optional embodiment.

[0061] in this case, Figure 3A This provides an optional visualization of the camera images captured during the determination of the eye's position relative to the mobile device. This allows the patient to examine the captured eye position within the captured images, enabling the capture of a new image or termination of the process if the eye recognition is incorrect.

[0062] Figure 3BThe display of gaze visualization 20 on display unit 12 is shown by way of example, with the aim of guiding the patient's gaze to gaze visualization.

[0063] Figure 3C The display of light visualization 16 on display unit 12 is illustrated by way of example during the capture of at least one image of an eye by means of camera 14. Therefore, the corneal reflection in each eye can be determined from the captured image. In this case, gaze visualization 20 can be displayed directly before and / or during the display of light visualization 16 to prevent the patient from taking their gaze away from the gaze visualization. As indicated by the dashed lines, in this case, camera 14, gaze visualization 20, and light visualization 16 can be pre-arranged relative to each other, and this can be used to determine the position of the corresponding corneal reflection.

[0064] The light visualization can be briefly displayed immediately after the gaze visualization (e.g., 100 ms to 500 ms thereafter), and one or more images can then be immediately captured by a camera. Depending on the quality, video frames can also be used for this purpose; however, it is assumed that photographs can provide a higher image quality than video frames. The very fast display of the light visualization 16 achieves this by keeping the eye's gaze still aligned with the gaze visualization and not yet directed to the pattern now visible on the display unit.

[0065] Figure 3D A schematic side view of the mobile device 10 relative to the eye is shown, which can be adapted to determine eye alignment. In this case, the lines visualize the direction vectors from the eye to the light visualization 16, to the gaze visualization 209, and to the camera 14. These direction vectors can be incorporated into the determination of the corresponding eye alignment to take any parallax into account.

[0066] Figure 3E An image of the eye 24 of patient 22 is shown by way of example, in which the corneal reflection 26 of the light visualization 16 is identifiable. This image may optionally be collected from captured images. Here, it is evident that the light visualization 16 is not located at the pupil center and / or iris center indicated by the intersecting lines, but rather has a horizontal and vertical offset from said pupil center and / or iris center.

[0067] This can be used as a basis for determining the vertical and / or horizontal displacement of the eye's line of sight from the pupil axis. This can optionally be calculated by the mobile device 10.

[0068] exist Figure 3F The text explains in more detail, in an exemplary and illustrative manner, that in Figure 3DThe diagram illustrates the distance and direction vectors. In this case, line 300 indicates the connection vector between the center of the light visualization or another marker point and the eye 24 (e.g., the center of the pupil and / or the center of the iris), and line 302 indicates the connection vector between the camera 14 and the eye 24. Arrow 304 indicates the distance of the eye 24 from the front side 10a of the mobile device 10 in the z-direction, and arrow 306 indicates the distance of the eye from the camera 14 in the y-direction. Alternatively, another distance in the x-direction between the camera 14 and the eye 24 can be determined. In this case, the eye can be specified to be centered relative to the camera, i.e., the distance from the bridge of the nose to the camera in the x-direction is approximately zero. The result could be that the x-distance of one eye is approximately between 3.15 cm and -3.15 cm.

[0069] Optionally, the patient can be encouraged to position and align the mobile device at a predetermined location such that the distance from the eye 24 to the camera 14 in the y-direction is approximately the same as the distance to the center of the light visualization 16. For this purpose, the y-distance from the eye to the camera can optionally be approximately 2 cm. The predetermined position of the eye can optionally be approximately 40 cm in front of the mobile device.

[0070] To determine the gaze direction (i.e., line of sight) of eye 24, the corresponding eye position relative to the mobile device is important. In this case, the three-dimensional position of the eye relative to the mobile device can optionally be calculated as follows:

[0071] Variant A) A depth map is created based on a depth sensor installed in the mobile device, and optionally, this depth map is adapted in terms of perspective to a portion of the image from camera 14. The position of a single eye can be captured in a 2D projection onto the captured image using camera-based face recognition (algorithmic solution). Based on the 2D position identified in the captured image, the distance of the eye in the z-direction (=> the z-component in the 3D position) can be collected from the depth map. Based on the camera's inherent calibration and the 2D position of the single eye in the camera image, the x and y components of the 3D position of the single eye can be calculated.

[0072] Variant B, described below, is similar to variant A in many respects, but instead, the x and y position components are also derived from the depth map. In this case, it can be considered whether the depth map has been distorted in perspective by adapting the image portion from the front camera. This can be corrected if necessary.

[0073] Furthermore, variant C, as an alternative to the absence of a depth sensor, will be described below. In this case, the z-distance is calculated based on the captured camera image. In this scenario, calibration may require an object of known size, which must be positioned at least once at a distance similar to the eye in the portion of the camera image (e.g., a credit card held in front of the eye or on the forehead). In this way, the current z-distance can be determined based on the size of the detected object. The identified distance can then be used to determine the pupil size and / or iris size of a single eye in the camera image. This identified distance can then be used as a reference parameter for determining the distance in the z-direction. The x and y components of the position can be determined according to variant A.

[0074] Furthermore, variant D will be described below as an alternative to the absence of a depth sensor and a reference object. In this case, the z-distance is calculated from the captured camera images, and the iris size and / or pupil size are used as reference parameters for determining the distance in the z-direction. The x and y components of the position can be determined according to variant A.

[0075] The position of the eye can optionally be determined periodically at defined time intervals. Optionally, this position is determined immediately before the display of light visualization. Optionally, the eye position can also be determined while issuing instructions to the patient to assist in the alignment of the mobile device, so as to always have accurate and up-to-date knowledge of any deviations from the positioning of the mobile device relative to the eye in the predetermined area.

[0076] Once the eyes are in a predetermined three-dimensional position relative to the mobile device, or once the mobile device is in a predetermined area relative to the eyes (in which case, a threshold for the three-dimensional position (x, y, z) can be predetermined), the gaze direction of the eyes can be further guided.

[0077] The perspective offset between the optical visualization on the display unit and the camera relative to the eye can be compensated for by having the patient center their gaze between the camera and the optical visualization. This can be achieved optionally by guiding the viewing direction using a gaze visualization (e.g., in the form of a flashing visual stimulus). The stimulus can optionally flash rapidly for a defined duration (e.g., 2 seconds). Subsequent steps can then be performed automatically.

[0078] The location of the flashing stimulus can optionally be calculated based on the camera position, the location of the light visualization on the display unit, and the position of the eye to be measured. Optionally, these locations form an isosceles triangle (see [reference]). Figure 3F In this context, the flashing stimulus is positioned at eye level and centered between the camera and the pattern.

[0079] The corneal reflectance of the corresponding eye for light visualization can be determined relative to the pupil center and / or iris center, for example, by means of algorithmic image processing, and the horizontal and / or vertical deviations between the corneal reflectance and the pupil center and / or iris center can be determined. The deviations can optionally be specified in millimeters and / or prism diopters. For this purpose, the Kappa angle, which specifies the difference between the optical axis and the line of sight, can now be calculated.

[0080] The corneal reflexes in both eyes are typically displaced slightly nasally by a Kappa angle in the horizontal direction. This offset can be taken into account when determining the horizontal / vertical displacement of the line of sight, where it is optionally assumed that at least one eye is free of strabismus.

[0081] Optionally, perspective correction can be performed on the line of sight based on a known perspective offset from the known three-dimensional relationships of the corresponding eye, the corresponding light visualization, and the corresponding camera.

[0082] List of reference numerals

[0083] 10 mobile devices

[0084] 10a Mobile Device Front Side

[0085] 12 display units

[0086] 14 cameras

[0087] 16 Light Visualization

[0088] 18 instructions

[0089] 20-Gaze ​​Visualization

[0090] 22 patients

[0091] 24 patients' eyes

[0092] 26 Corneal reflex

[0093] 200 Methods for determining eye alignment

[0094] Method steps 202 to 212

[0095] The connection vector between the center of 300 light visualization and the eye

[0096] 302 Connection vector between camera and eye

[0097] 304 z-distance between the front of the mobile device and the eye

[0098] The y-distance between the camera and the eye (306).

Claims

1. A computer-implemented method (200) for determining an eye alignment of eyes (24) of a patient (22), the method to be performed by a mobile device (10) having a display unit (12) and a camera (14) arranged on the same side of the mobile device (10) as the display unit (12), characterized in that the method (200) comprising: outputting (202) instructions (18) to the patient (22) for assisting the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined area relative to the eyes (24) of the patient (22) and face the eyes (24); determining (204) a position of the eyes (24) relative to the mobile device (10); displaying (206) a light visualization (16) on the display unit (12); capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the light visualization (16) displayed on the display unit (12) in each eye (24) of the patient (22) can be determined in the captured image; and determining (210) a position of the respective corneal reflection (26) in the eye (24) based on the at least one captured image and using the position of the respective corneal reflection (26) in the eye (24) for determining the eye alignment of the eyes (24).

2. The method (200) of claim 1, wherein, Determining the position of the respective corneal reflection (26) in the eye (24) comprises determining a relative position of the corneal reflection (26) relative to a pupil center and / or an iris center of the respective eye (24).

3. The method (200) of claim 1 or 2, wherein, Determining the eye alignment of the eyes (24) comprises determining a deviation of a line of sight from a pupil axis of each eye of the eyes (24).

4. The method (200) of any of the preceding claims, wherein, The light visualization (16) has a brightness and / or a size on the display unit (12) such that the light visualization (16) can be determined by performing an image evaluation of the at least one image with the mobile device (10).

5. The method (200) of any one of the preceding claims, further comprising: displaying (212) a fixation visualization (20) on the display unit (12) prior to and / or simultaneously with displaying (206) the light visualization (16) and capturing (208) the at least one image in order to direct a fixation gaze of the patient (22) onto the fixation visualization (20).

6. The method (200) of any of the preceding claims, wherein, Determining (202) the position of the eyes (24) relative to the mobile device (10) comprises determining the position of the eyes (24) relative to the camera (14) and / or relative to the fixation visualization (20).

7. The method (200) of any of the preceding claims, wherein, Outputting (202) the instructions (18) to the patient (22) is performed by means of the display unit (12) and optionally comprises a graphical user interaction.

8. The method (200) of any of the preceding claims, wherein, The method (200) is configured such that the method (200) allows for a self-test by the patient (24).

9. A computer program and / or a computer readable medium, characterized in that, The computer program and / or the computer readable medium comprises instructions which, when executed by a mobile device (10) having a display unit (12) and a camera (14) arranged at the same side of the display unit (12), cause the mobile device (10) to perform a method for determining an eye alignment (26) of eyes (24) of a patient (22), wherein the method (200) comprises: outputting (202) instructions (18) to the patient (22) for assisting the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined area relative to the eyes (24) of the patient (22) and face the eyes (24); determining (204) a position of the eyes (24) relative to the mobile device (10); displaying (206) a light visualization (16) on the display unit (12); capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the light visualization (16) displayed on the display unit (12) in each eye (24) of the patient (22) can be determined in the captured image; and determining (210) a position of the respective corneal reflection (26) in the eyes (24) based on the at least one captured image and using the position of the respective corneal reflection (26) in the eyes (24) to determine an eye alignment of the eyes (24).

10. A mobile device (10) for determining an eye alignment of eyes (24) of a patient (22), the mobile device (10) comprising: a display unit (12); and a camera (14) arranged at the same side of the mobile device (10) as the display unit (12); characterized in that the mobile device (10) is configured to perform the following steps: outputting (202) instructions (18) to the patient (22) for assisting the patient (22) in aligning the mobile device (10) relative to the eyes (24) of the patient (22) such that the display unit (12) and the camera (14) are positioned in a predetermined area relative to the eyes (24) of the patient (22) and face the eyes (24); determining (204) a position of the eyes (24) relative to the mobile device (10); displaying (206) a light visualization (16) on the display unit (12); capturing (208) at least one image of the eyes (24) by means of the camera (14) such that a corneal reflection (26) of the light visualization (16) displayed on the display unit (12) in each eye (24) of the patient (22) can be determined in the captured image; and determining (210) a position of the respective corneal reflection (26) in the eyes (24) based on the at least one captured image and using the position of the respective corneal reflection (26) in the eyes (24) to determine an eye alignment of the eyes (24). ​ 11. The mobile device (10) of claim 10, wherein, The mobile device (10) is embodied as a smartphone or a tablet computer.

12. The mobile device (10) according to claim 10 or 11, wherein The display unit (12) and the camera (14) are arranged on a side of the mobile device (10) facing towards the eyes (24) of the patient (22) when said mobile device (10) is used by the patient (22).

13. The mobile device (10) according to any one of claims 10 to 12, wherein, The display unit (12) is embodied as a display, and optionally as a touch screen.

14. The mobile device (10) according to any one of claims 10 to 13, wherein, The mobile device (10) is configured to allow determining an eye alignment of the eyes (24) of the patient (22) within the scope of a self-test of the patient (22).

Images