Method and system for operating an optometric device

Abstract

The invention relates to a computer-implemented method for operating an optometry device, comprising the steps of:-generating a first data set comprising information about a distance of a user of the optometry device to an object; -tuning a tunable lens of the optometry device as a function of the distance of the user of the optometry device from the object based on the first data set. The computer-implemented method of the invention is characterized in that the first data set is generated by using a LiDAR sensor which measures the distance of the user of the optometry device to the object by evaluating the scene represented by the first data set.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for operating an optometry device according to the preamble of claim 1 and to a system comprising an optometry device according to the preamble of claim 10 . Background Art

[0002] US2022 / 0146819 A1 discloses a device and method for visual depth prediction. The device includes a camera assembly and a controller. The camera assembly is configured to capture images of both eyes of a user. Using the captured images, the controller determines the position of each pupil of each eye of the user. The determined pupil positions and the captured images are used to determine eye tracking parameters, which are used to calculate the value of an eye tracking function. Using the calculated values ​​and a model that maps the eye tracking function to visual depth, the visual depth of the user is determined.

[0003] EP 3 498 149 A1 discloses an optoelectronic binocular instrument for automatic correction of presbyopia and a method for binocular correction of presbyopia. The instrument has two optoelectronic lenses and a capture subsystem for taking images of the eye. With the help of pupil tracking, which performs processing on the image of the eye, the system determines the distance at which the subject is looking. Pupil tracking works at very high speeds using a high-performance graphics processor and a highly parallelized algorithm for pupil tracking. The method consists of two stages. In the first stage, calibration is done, the subject is asked to look at a target at different distances, and the size and position of the pupil are measured. In the second stage, the correction is performed by the instrument, the system continuously captures and processes images to calculate the correction to be applied, and finally corrects presbyopia by applying the correction.

[0004] US 7,656,509 B2 discloses devices for determining distance. An object is being looked at by a user presenting an electro-activated lens. Once the distance is determined, these devices can change the optical power of the electro-activated lens to ensure that the object is properly focused. Optical ranging is a possible means of performing this task. An active rangefinder can emit optical radiation directed at an object from a transmitter. The optical radiation can then be reflected by the object. The reflected optical radiation can then be received by a suitable receiver. The received optical radiation can then be processed by suitable circuitry to determine the distance to the object. A passive rangefinder works without a transmitter. Instead, a suitable receiver receives ambient light sources from the object. The received light can then be processed by suitable circuitry to determine the distance to the object.

[0005] US11 221 488 B1 discloses a pair of glasses, which includes one or more adjustable lenses, each adjustable lens is configured to align with a corresponding eye of a user. These adjustable lenses include a concave liquid crystal adjustable lens stacked with a non-liquid crystal adjustable lens (e.g., a fluid-filled lens or an Alvarez lens). The concave adjustable lens includes an electrically modulated optical material, such as one or more liquid crystal cells. The liquid crystal cell includes an electrode array extending in one, two, three, four or more directions. The control circuit system can apply a control signal to the electrode array in each liquid crystal cell to produce a desired phase profile. Each lens can be concave so that the portion of the lens within the user's visual field exhibits a different phase profile than the portion of the lens outside the user's visual field.

[0006] US2020 / 379214 A1 discloses an augmented reality (AR) device, which includes a zoom lens, the focal length of which can be changed by adjusting the refractive power and adjusting the position of the focus adjustment area of ​​the zoom lens according to the viewing direction of the user. The AR device can use an eye tracker to obtain an eye vector indicating the viewing direction of the user, adjust the refractive power of the first focus adjustment area of ​​the first zoom lens to change the focal length for displaying a virtual image, and complementarily adjust the refractive power of the second focus adjustment lens relative to the refractive power of the adjusted first focus adjustment area.

[0007] Hasan, N., Karkhanis, M., Khan, F., Ghosh, T., Kim, H., and Mastrangelo, CH (2017), Adaptive Optics for Autofocusing Eyeglasses, in Optics InfoBase Conference Papers (https: / / doi.org / 10.1364 / AIO.2017.AM3A.1), describes implementations of adaptive eyeglasses designed to restore the ability to accommodate lost due to age-related presbyopia. The adaptive variable-power eyepiece is driven by a battery-powered microcontroller system that measures the distance from the observer to the object and combines it with the observer's eye prescription to produce a sharp image at any object distance.

[0008] Mompeán, J., Aragón, JL, and Artal, P. (2020) Portable device for presbyopia correction with optoelectronic lenses driven by pupil response, Scientific Reports, 10(1) (https: / / doi.org / 10.1038 / s41598-020-77465-5), describes a portable device that has been developed and built to dynamically and automatically correct presbyopia with the help of a pair of optoelectronic lenses driven by pupil tracking. The system is fully portable and provides a high defocus correction range of up to 10D. The glasses are controlled and powered by a smartphone. To achieve a truly real-time response, the image processing algorithms have been implemented in OpenCL and run on the GPU of the smartphone. To validate the system, different visual experiments were conducted in presbyopic subjects. Visual acuity remained almost constant over a distance range of 5m to 20cm.

[0009] Padmanaban, N., Konrad, RK, and Wetzstein, G. (2019) in Autofocals: Evaluating gaze-contingent eyeglasses for presbyopes, in ACM SIGGRAPH 2019 Talks, SIGGRAPH 2019, Association for Computing Machinery, Inc (https: / / doi.org / 10.1145 / 3306307.3328147), describes the estimation of the depth of the gazed object, which is estimated dynamically via sensor fusion of four "raw" inputs: two gaze tracking cameras, a scene-facing depth camera, and the user's interpupillary distance (IPD). A binocular eye tracker estimates vergence distance at 120 Hz. However, small errors in the gaze direction estimate can introduce significant bias in the estimated vergence. Although the depth sensor runs at only 30 Hz, it, together with the gaze direction, compensates for the bias in the vergence measurement. The authors developed a custom sensor fusion algorithm to balance the accuracy and speed of the vergence estimation pipeline.

[0010] Jarosza J., Molliexa N., Lavignea Q., and Berge B. (2019) An original low-power opto-fluidic engine for presbyopia-correcting adaptive eyeglasses, in Ophthalmic Technologies XXIX, edited by Fabrice Manns, Per G. Soderberg, Arthur Ho, Proceedings of the SPIE Conference, Vol. 10858, 1085824, doi: 10.1117 / 12.2507816, describes a concept for presbyopia-correcting adaptive eyeglasses for people who do not accept corrective solutions. The proposed glasses automatically provide clear vision at all distances and are characterized by enhanced field of view and high optical quality. The adaptive technology relies on original fluid-filled lenses whose focusing power is set by a low-power microfluidic pump inserted in the temples of the glasses. A LiDAR sensor is used to measure the focusing distance.

[0011] Tunable lenses in the prior art typically include an IR camera for scene detection and an eye tracking device for depth recognition. Tunable lenses in the prior art typically have limitations in depth resolution and distance estimation accuracy. In addition, the tuning of known tunable lenses is achieved after determining the user's gaze point. Therefore, the tuning of the tunable lens lags behind the user's gaze process. In addition, the scene context is typically not considered when operating the tunable lens.

[0012] Problem to be solved

[0013] Therefore, in particular in view of US Pat. No. 11 221 488 B1 and the publication by Jarosza et al., it is an object of the present invention to provide a computer-implemented method and system that overcome the above-mentioned limitations of the prior art.

[0014] It is a specific object of the present invention to provide a computer implemented method and system with a more accurate and faster way of measuring distance to enable more accurate and natural tuning of optometry equipment. Summary of the invention

[0015] The problem is solved by a computer-implemented method and a system including an optometry device.

[0016] In a first aspect, the present invention relates to a computer-implemented method for operating an optometry device, the method comprising the following steps:

[0017] - generating a first data set comprising information about the distance of the object from the user of the optometry device,

[0018] - tuning a tunable lens of the optometry device according to the distance of the object from a user of the optometry device based on the first data set.

[0019] The computer-implemented method is characterized in that the first data set is generated by using a LiDAR sensor that measures the distance of the user of the optometry device to the object by evaluating the scene represented by the first data set.

[0020] As commonly used, the term "computer-implemented method" refers to a method involving at least one device, in particular a computer, or a plurality of devices, in particular connected via a computer network. The plurality of devices may be connected via a network using at least one connection interface at any of the plurality of devices. The computer-implemented method may be implemented as at least one computer program that may be provided on a storage medium carrying the computer program, whereby at least one step of the computer-implemented method is performed by using the at least one computer program. Alternatively, the at least one computer program may be accessed by a device that may be adapted to perform the method via a network, such as via an intranet or via the Internet.

[0021] The term "optometry device" refers to a device that processes light waves. Within the scope of the present invention, the term "optometry device" refers to at least eyeglass lenses, binoculars, contact lenses or any other device that corrects or detects eye problems of a user.

[0022] The term "dataset" in the context of the present invention describes an item, such as a numeric or alphanumeric item, comprising at least one piece of information. The data may be provided in a machine-readable form, such that the data may be an input or output of a machine learning model, in particular an input or output of any neural network comprised by the machine learning model, or the data may be an input or output of a computer-implemented method. In particular in view of the present aspect, the term "first data set" refers to the output of a LiDAR sensor. The output may be represented by a point cloud or digital information about the distances of multiple objects in a scene.

[0023] As commonly used, the term "tuning" or any grammatical variations thereof refers to the process of adjusting the optical power of an optometric device.

[0024] The term “optical power” is a general term for the spherical vertex power of a spectacle lens, which brings a paraxial parallel beam to a single focus, and the cylindrical vertex power, which brings a paraxial parallel beam to two separate focal lines at right angles to each other (DIN ISO 13666:2019, section 3.10.2).

[0025] The term "tunable lens" refers to a light focusing device (in the form of a fluid, gel, or solid) that can dynamically modulate its focal length using input electrical energy. Thus, in a tunable lens, some internal property that changes the shape of the light wavefront can be electrically modified.

[0026] An example of a tunable lens is a shape-changing lens based on a combination of an optical fluid and a polymer film. The core element consists of a container filled with an optical fluid and sealed with a thin elastic polymer film. The optical fluid can be a transparent liquid with specific optical properties (such as refractive index). A ring pushed to the center of the membrane will shape the tunable lens. The deflection of the membrane and the radius of the lens can be changed by pushing the ring against the membrane, by applying pressure to the outside of the membrane, or by pumping liquid into or out of the container. Other examples of tunable lenses are Alvarez lenses, which shift a pair of complementary cubic phase plates relative to each other to change the optical power; and lenses using liquid crystal technology, which change the orientation of the liquid crystals as an electric current changes, thereby changing the refraction of light.

[0027] The term "LiDAR" is an acronym for "light detection and ranging" or "laser imaging, detection, and ranging." LiDAR sensors are used to determine distance by pointing a laser at an object or surface and measuring the time it takes for the reflected light to return to a receiver. In the present invention, it is used for distance measurement for mobile applications.

[0028] As commonly used, the term "evaluation" or any of its grammatical variations refers to the process of analyzing a scene. The term "scene" in the context of the present invention describes the field of view of a person or the field of view of a digital device (such as a LiDAR sensor or a camera). In the context of the present invention, a scene is represented by an image, a video, or a dataset. As used herein, the term "image"

[0029] Refers to a single picture, especially a picture of an environment. As used herein, the term "video"

[0030] Refers to multiple images of the environment.

[0031] By means of the computer-implemented method, which comprises the further step of measuring the distance from the user of the optometry device to the object using a LiDAR sensor, and tuning the tunable lens of the optometry device based on the distance from the user of the optometry device to the object, a more accurate distance measurement is achieved. Thus, a more accurate tuning of the tunable lens of the optometry device is achieved. This results in a higher wearing comfort for the user of the optometry device.

[0032] In a particularly preferred embodiment, the computer-implemented method comprises the step of generating the second data set by recording images of the scene by a camera.

[0033] The term "second data set" refers to the output of the camera. The output may be represented by an image or a video. The image or video includes a scene having at least one identifiable object. In the present invention, the scene of the "second data set" is recorded substantially in parallel with the scene of the "first data set". In other words, the camera and the LiDAR sensor observe the same object in the scene, but provide different data sets representing the same scene.

[0034] In addition, the first data set and the second data set include different data formats and different acquisition parameters. The first data set recorded by the LiDAR sensor may include a point cloud of the scene, the point cloud having points of different distances identified within the scene. The first data set may be recorded at a resolution of 940 pixels × 560 pixels and a frame rate of 30fps. The second data set recorded by the camera may include at least one image of the same scene recorded by the LiDAR sensor. The second data set may be recorded at a resolution of 1920 pixels × 1080 pixels and a frame rate of 30fps. Therefore, the resolution of recording the second data set may be higher than the resolution of recording the first data set. Alternatively, the resolution of recording the first data set may be higher than the resolution of recording the second data set.

[0035] If the resolution of the first data set is not equal to the resolution of the second data set, the data set with the larger resolution can be scaled down to the smaller resolution of the other data set. Alternatively, the data set with the smaller resolution can be interpolated to the other data set with the higher resolution. In the above example, the resolution of the second data set recorded by the camera can be scaled down to the resolution of the first data set recorded by the LiDAR sensor. Therefore, the resolution of the second data set will be scaled down to 940 pixels × 560 pixels.

[0036] In addition, the field of view of the first data set may be at least slightly wider than the field of view of the second data set. In this case, the field of view of the first data set is cropped to the field of view of the second data set. Alternatively, the field of view of the second data set may be at least slightly wider than the field of view of the first data set.

[0037] If the frame rate of the first data set is not equal to the frame rate of the second data set, the data set with the larger frame rate can be scaled down to the smaller frame rate of the other data set. In the above example, the frame rate of the second data set recorded by the camera is equal to the frame rate of the first data set recorded by the LiDAR sensor. Therefore, the frame rate of the second data set will not be scaled down.

[0038] The second data set of the computer-implemented method enables an evaluation of the recorded scene. Thus, the further step of tuning the tunable lens of the optometry device is based on and optimized by the information of the recorded scene. This makes the focus adjustment of the tunable lens of the optometry device more natural.

[0039] In a particularly preferred embodiment, the computer-implemented method comprises the step of generating the second data set by recording images of the scene by means of an RGB camera.

[0040] The term "RGB camera" refers to a camera that provides an image matrix. Each point in the grid is assigned a color value. The resolution of the image is calculated based on the number of rows and columns. The output of the RGB camera can be represented by an image or a video. The image or video includes a scene with at least one identifiable object. In the present invention, the scene of the "second data set" is recorded in parallel or substantially in parallel with the scene of the "first data set". In other words, the RGB camera and the LiDAR sensor observe the same objects in the scene, but provide different data sets representing the same scene.

[0041] The second data set recorded by the RGB camera enables an evaluation of the recorded scene. Thus, the further step of tuning the tunable lens of the optometry device is based on and optimized by the information of the recorded scene. This makes the focus adjustment of the tunable lens of the optometry device more natural.

[0042] In a particularly preferred embodiment, the computer-implemented method comprises the further step of identifying the object by evaluating the scene represented by the second data set.

[0043] As commonly used, the term "identification" or any grammatical variation thereof refers to an identification process after recording an image of a scene. At the end of the identification process, objects can be identified and selected for tuning the optometry device based on the distance of the identified and selected objects to the user of the optometry device.

[0044] The further step of identifying the object enables the computer-implemented method to not only recognize but also identify the object within the recorded scene. The further step of identifying the object thus brings about the further step of tuning the tunable lens of the optometry device based on the known and identified object. The object in the scene is identified by recording an image with an RGB camera and the distance of the object to the user of the optometry device is determined by the data measured by the LiDAR sensor and the distance is assigned to the object. This makes the focus adjustment of the tunable lens of the optometry device more natural and the wearing comfort of the user of the optometry device better.

[0045] In a particularly preferred embodiment, the computer-implemented method is characterized in that identifying the object comprises the following sub-steps:

[0046] - identifying a plurality of objects in an image of a scene represented by a second dataset, and

[0047] - generating an identification probability factor for each identified object of the plurality of objects using the first database, and

[0048] - assigning an identification probability factor to each of the plurality of objects based on the first database, and

[0049] - comparing the identification probability factor of each identified object of the plurality of objects with a predefined identification probability threshold, and

[0050] - generating a selection probability factor for each identified object in the plurality of objects having an identification probability factor greater than the predefined identification probability threshold using a second database, and

[0051] - comparing the selection probability factor of at least each identified object of the plurality of objects having an identification probability factor greater than the predefined identification probability threshold with the predefined selection probability threshold, and

[0052] - determining a distance of the user of the optometry device to the identified object having the largest selection probability factor, wherein the selection probability factor of the identified object is greater than the predefined selection probability threshold.

[0053] In a first step of the identification process, a plurality of objects in a recorded image of a scene may be identified. As commonly used, the term "identification" or any grammatical variant thereof refers to an identification sub-step after recording an image of a scene. At the end of said first step of the identification process, it is known whether an object is present within the scene. Identifying an object within the scope of the present invention may be achieved by evaluating a first data set or preferably a second data set.

[0054] In a further step of the identification process, an identification probability factor is generated for each identified object of the plurality of objects using the first database, and at the end of said further step of the identification process, the identity of the identified objects in the scene is known based on a certain probability.

[0055] The term "database" in the context of the present invention describes a collection of numerical methods, a collection of machine learning models, or a collection of algorithms based on artificial intelligence. In the present invention, the term "first database" describes a collection of trained machine learning models for identifying objects within a scene context, wherein the trained computer learning model includes at least one identification probability factor. The identification probability factors of objects can be stored in a table within the first database or in any other form of information storage. Examples of the first database are TensorFlow, Image Net, or PyTorch.

[0056] The machine learning model has been trained to recognize and identify at least one object and the context in which the object is detected. For example, the object may be a computer keyboard and the context may be a computer keyboard on a desk. The computer keyboard and the desk together with the surrounding environment do represent a scene. As used herein, the term "trained machine learning model" refers to a model of a machine learning model, specifically including at least one neural network that is trained, in particular trained to determine at least one identification probability factor.

[0057] The term "identification probability factor" refers to a certain probability generated by the trained machine learning model by analyzing the objects identified in the second data set. The aforementioned example includes a computer keyboard on a desk. In this example, the trained machine learning model generates an identification probability factor that the identified object is a computer keyboard on a desk. The identification probability factor can contain values ​​in the range of 0 to 1, including 0 and 1, equivalent to 0% to 100%. An identification probability factor of 0 is equivalent to 0%, and means that the first database does not know which object is identified in the context. Therefore, there is no such data available for training the machine learning model to identify, for example, a computer keyboard on a desk. In this case, the machine learning model can be trained in a further step. An identification probability factor of 1 is equivalent to 100%, and means that the first database does know which object is identified in the context with the highest possible probability. Therefore, there is enough data available for training the machine learning model to identify, for example, a computer keyboard on a desk. In this case, there is no need to train the machine learning model in a further step. The identification probability factor preferably includes the identification probability of at least one object within the context, or alternatively includes an identification probability factor of at least one object without context.

[0058] The term "generating" or any grammatical variation thereof refers to the process of generating certain information within the scope of the present invention. This may refer to, for example, generating identification probability factors or generating data sets. Information may be generated by calculating, recording, extracting, assigning, deriving, processing, loading, predicting, or any other possible information generation step.

[0059] In a further step of the identification process, an identification probability factor is assigned to each object of the plurality of objects based on the first database. At the end of said further step of the identification process, identification probability factors are assigned to the identified objects in the scene.

[0060] The term "assignment" or any grammatical variation thereof refers to the process of merging one piece of information with a second piece of information. For example, the second data set may provide the presence of an object. The first data set generates an identification probability factor for the object. By assigning a certain percentage of a particular identification probability factor to the identified object, the computer-implemented method knows that, for example, a computer keyboard on a table will be identified within the scene with a probability of the certain percentage. The certain percentage is in the range of 0% to 100%.

[0061] In a further step of the identification process, the identification probability factor of each identified object of the plurality of objects is compared to a predefined identification probability threshold. At the end of the further step of the identification process, all objects having an identification probability factor below the predefined identification probability threshold are ignored for the further step. If no identified object includes an identification probability factor greater than the predefined identification probability threshold, i) the predefined identification probability threshold may be lowered to the highest assigned identification probability factor of the identified objects to ensure further processing of the computer-implemented method, or ii) the computer-implemented method stops at this step without tuning the tunable lens and proceeds to the next scenario to tune the tunable lens of the optometry device. Alternatively, if no object can be identified, the tunable lens may be tuned to a default distance based on the raw data of the first data set, for example, to infinity, or to another distance. If more than one identified object includes an identification probability factor greater than the predefined identification probability threshold, all such objects proceed to the next step of the identification process.

[0062] The term "predefined identification probability threshold" refers to a filtering step used for further processing of the identified objects. The predefined identification probability threshold may include values ​​in the range of 0 to 1, including 0 and 1, equivalent to 0% to 100%. In a preferred embodiment, the value of the predefined identification probability threshold is between 0.5 and 0.9, equivalent to 50% to 90%. In a particularly preferred embodiment, the value of the predefined identification probability threshold is between 0.6 and 0.8, equivalent to 60% to 80%. In a most preferred embodiment, the value of the predefined identification probability threshold is 0.7, equivalent to 70%.

[0063] In a further step of the identification process, a selection probability factor is generated for each identified object of the plurality of objects or at least for all objects having an identification probability factor equal to or greater than an identification probability threshold using the second database. At the end of said further step of the identification process, an identified object is selected, said selected object comprising a certain selection probability observed by a user of the optometry device.

[0064] In the present invention, the term "second database" describes a collection of trained machine learning models for selecting objects within a scene context, wherein the trained machine learning models include at least one selection probability factor. The selection probability factors of the objects may be stored in a table within the second database or in any other form of information storage. The selection probability factors of the second database are selection probabilities based on users of the optometry device.

[0065] Furthermore, the second database includes individual characteristics of the users. For example, a user over retirement age shows a smaller selection probability factor for switching the field of view to the computer compared to a younger user who works with the computer every day. A younger user will show a larger selection probability factor for the computer compared to a user over retirement age. Alternatively, the user's past behavior (which is taken from the second database) shows that the user never looks at the computer. Therefore, the selection probability factor for the computer for this user is small. Other parameters that influence the visual behavior (such as time of day, day of the week, etc.) and therefore the selected object can also be included in the second database. Examples of second databases are DeepGaze or SCEGRAM.

[0066] The machine learning model has been trained to select at least one object from a context in which the object is detected. For example, the objects may be a computer keyboard and a computer mouse, and the context may be a computer keyboard and a computer mouse on a desk. As used herein, the term "trained machine learning model" also refers to a model of a machine learning model, specifically including at least one neural network that is trained, in particular trained to determine at least one selection probability factor.

[0067] The term "selection probability factor" refers to a certain probability generated by a trained machine learning model by analyzing the (multiple) objects identified and identified in the second data set. The aforementioned examples include a computer keyboard and a computer mouse on a desk. In this example, the trained machine learning model generates a selection probability factor for the identified and identified objects. In other words, the machine learning method generates values ​​including selection probabilities for multiple objects within the scene. The selection probability factor can contain values ​​in the range of 0 to 1, including 0 and 1, equivalent to 0% to 100%. A selection probability factor of 0 is equivalent to 0%, and means that based on the second database, the user will focus on the selection probability of the object identified and identified in the context does not exist. A selection probability factor of 1 is equivalent to 100%, and means that based on the second database, the user will focus on the selection probability of the object identified and identified in the context is 100%. The selection probability factor preferably includes the selection probability of at least one object within the context, or alternatively includes the selection probability of at least one object without context. Therefore, the selection probability factor of a particular object is a measure of the probability or possibility that the user will focus on the particular object.

[0068] In a further step of the identification process, the selection probability factor of at least each identified object of the plurality of objects having an identification probability factor greater than the predefined identification probability threshold is compared with the predefined selection probability threshold. At the end of said further step of the identification process, all objects having a selection probability factor below the predefined selection probability threshold are ignored for consecutive further steps.

[0069] If none of the identified objects include a selection probability factor greater than a predefined selection probability threshold, i) the predefined selection probability threshold may be lowered to the highest assigned selection probability factor of the identified and identified objects to ensure further processing of the computer-implemented method, or ii) the computer-implemented method stops at this step without changing the focus of the tunable lens and proceeds to the next scene to tune the tunable lens of the optometry device. If more than one identified object includes a selection probability factor greater than the predefined selection probability threshold, the object with the greater selection probability factor will proceed to the next step of the identification process.

[0070] The term "predefined selection probability threshold" refers to a filtering step used for further processing of the identified objects. The predefined selection probability threshold may comprise a value in the range of 0 to 1, inclusive, corresponding to 0% to 100%. In a preferred embodiment, the value of the predefined selection probability threshold is between 0.5 and 0.9, corresponding to 50% to 90%. In a particularly preferred embodiment, the value of the predefined selection probability threshold is between 0.6 and 0.8, corresponding to 60% to 80%. In a most preferred embodiment, the value of the predefined selection probability threshold is 0.7, corresponding to 70%.

[0071] In a further step of the identification process, a distance of the user of the optometry device to the identified object with the largest selection probability factor is determined based on the step of comparing the selection probability factor of at least each identified object of the plurality of objects having an identification probability factor greater than a predefined identification probability threshold with a predefined selection probability threshold and selecting the object with the highest selection probability factor. At the end of said further step of the identification process, the computer-implemented method is capable of tuning a tunable lens of the optometry device according to the distance of the user of the optometry device to the object with the highest selection probability factor based on the second data set.

[0072] The further step of identifying the object enables the computer-implemented method to not only recognize but also identify the object within the recorded scene. Thus, the further step of identifying the object brings about the further step of tuning the tunable lens of the optometry device based on the known and identified object. This makes the focus adjustment of the tunable lens of the optometry device more natural and more accurate, and makes the wearing comfort of the user of the optometry device better. Furthermore, the tuning of the tunable lens can be achieved before the user switches the line of sight to another object.

[0073] In a particularly preferred embodiment, the computer-implemented method is characterized in that the identification probability factors stored in the first database and the selection probability factors stored in the second database are determined using a machine learning algorithm.

[0074] The further step of identifying the object enables the computer-implemented method to load the identification probability factor from the first database or the selection probability factor from the second database without the need to calculate a new identification probability factor or a new selection probability factor. This enables faster focus adjustment of the tunable lens of the optometry device and better wearing comfort for the user of the optometry device. Furthermore, tuning of the tunable lens can be achieved before the user switches his sight to another object.

[0075] In a particularly preferred embodiment, the computer-implemented method is characterized by generating a third data set comprising visual information of a user of the optometry device, predicting the user's next visual sight based on the third data set, and tuning a tunable lens of the optometry device based on the predicted next visual sight of the user.

[0076] In a specific embodiment of this aspect, the term "third data set" refers to the output of the eye tracking sensor. The output can be represented by an RGB image or numerical information about the user's glance and / or gaze point in the context of the scene. The term "saccade" refers to the movement of at least one eye between two or more gaze stages in the same direction. The term "gaze" or "gaze point" refers to the movement behavior of the eyes. During the "gaze", the spatial position of the gaze continues to be at a single location. In other words, if the gaze is achieved, the eyes do not move in a recognizable way.

[0077] The term "visual information" refers to the field of view of a user of an optometry device and may refer to changes in a person's line of sight. Based on DIN ISO 13666:2019, Section 3.2.24, the term "line of sight" refers to the path from a point of interest in object space (i.e., the point of gaze (or fixation point)) to the center of the entrance pupil of a person's eye, and further includes the continuation in image space from the center of the exit pupil to the retinal fixation point of the person's eye (usually the fovea). The term "predicted next visual" refers to the estimated next field of view of a user of an optometry device.

[0078] The term "next fixation point" in the context of the present invention describes the end of a glance and the start of a new fixation. The term "predicted next fixation point" refers to the next fixation point generated based on the third data set.

[0079] A third data set of the computer-implemented method recorded by the eye tracker sensor enables an evaluation of the user's eye movements based on the recorded scene represented by the first and second data sets. Thus, a further step of tuning the tunable lens of the optometry device is based on and optimized by the predicted next gaze point of the user. This enables faster focus adjustment of the tunable lens of the optometry device and better wearing comfort for the user of the optometry device. Furthermore, tuning of the tunable lens can be achieved before the user switches his gaze to another object.

[0080] In a particularly preferred embodiment, the computer-implemented method is characterized in that predicting the user's next glance comprises the following steps:

[0081] - determining a current gaze point of a user of the optometry device based on the third data set, and

[0082] - assigning the object based on the second data set and the distance of the object to the user of the optometry device based on the first data set to the current gaze point of the user of the optometry device, and

[0083] - tracking a current gaze point of the user of the optometry device based on the third data set until a next glance of the user of the optometry device is identified by detecting a change in gaze position, and

[0084] - predicting a next gaze point of the user of the optometry device based on the current gaze point of the user of the optometry device and the identified next glance of the user of the optometry device, and

[0085] -Tuning optometry equipment based on the predicted next fixation point.

[0086] In a first step of the prediction process, a current gaze point of a user of the optometry device is determined based on the third data set.At the end of said first step of the prediction process, a gaze point is determined for the start of a new glance (indicated by time t=0).

[0087] The term "current fixation point" refers to i) the fixation point that has been established when the prediction process starts, or ii) the fixation point that is detected after the prediction process starts.

[0088] In a further step of the prediction process, the object identified and selected based on the second data set and the distance of the object to the user of the optometry device based on the first data set are assigned to the current gaze point of the user of the optometry device. Alternatively, by a further step of the prediction process, the current gaze point is assigned to the object identified and selected based on the second data set and the distance of the user of the optometry device to the object based on the first data set.

[0089] In a further step of the prediction process, the current gaze point of the user of the optometry device is tracked based on the third data set until a next glance of the user of the optometry device is identified by detecting a change in gaze position. At the end of said further step of the prediction process, the start of a new glance is detected.

[0090] The term "tracking" or any grammatical variation thereof refers to the process of identifying changes in information within the scope of the present invention. This may refer to, for example, tracking a scene or visual position. Tracking changes in information may be done by observing, detecting, analyzing, searching, or any other possible step of identifying changes in information.

[0091] In a further step of the prediction process, a next gaze point of the user of the optometry device is estimated based on the current gaze point of the user of the optometry device and the identified next glance of the user of the optometry device. At the end of said further step of the prediction process, the next gaze point of the user of the optometry device is predicted.

[0092] The term "prediction" or any grammatical variation thereof refers to the process of estimating information within the scope of the present invention. This may refer to predicting a scene or a fixation point, for example. The estimation of information may be performed by calculating, interpolating, comparing, guessing, or any other step of estimating information.

[0093] In a further step of the prediction process, the optometry device is tuned according to the predicted next fixation point.

[0094] The further step of identifying the object enables the computer-implemented method to estimate the next gaze point of the user of the optometry device. This enables faster focus adjustment of the tunable lens of the optometry device and better wearing comfort for the user of the optometry device. Furthermore, tuning of the tunable lens can be achieved before the user switches his gaze to another object.

[0095] In a particularly preferred embodiment, the computer-implemented method is characterized in that predicting the next fixation point comprises at least one of the following steps:

[0096] - determining the gaze direction of the next identified glance and predicting the next gaze point based on the user's previous gaze information;

[0097] - determining a gaze direction of the identified next glance and predicting a next fixation point based on a next selectable object within the determined gaze direction;

[0098] - determining a gaze direction of the identified next glance and predicting a next fixation point based on an object within the determined gaze direction, the object comprising the highest selection probability factor;

[0099] - using a predefined next scene of the second database and assigning a distance of an object to a user of the optometry device based on the first data set and adjusting the estimate of the next gaze point based on the predefined next scene;

[0100] - estimating a next scene by generating a fourth data set comprising brain electrical activity information of a user of the optometry device, and adjusting an estimate of the next fixation based on the estimated next scene;

[0101] - estimating a next scene by generating a fifth data set comprising inertial measurement information of the optometry device (110, 710), and adjusting an estimate of the next fixation point (365, 465, 565, 665) based on the estimated next scene.

[0102] The step of predicting the next gaze point may be implemented by determining the gaze direction of the identified next glance and predicting the next gaze point based on the user's previous glance information.

[0103] The term "gaze direction" refers to the spatial localization detection of the current glance. Starting from the current gaze point, the next glance can move in each spatial direction, for example, in the northeast direction. Gaze direction can be specified in the form of celestial direction, global coordinate system or local coordinate system or by spatial direction.

[0104] The term "previous visual information" refers to a sample of previous visual information of a user of the optometry device or a standard user of the optometry device. The previous visual information may include data of a glance distance length, a glance duration, a fixation point, and a fixation duration. The previous visual information is generated and stored in a third data set.

[0105] Starting from the current fixation point and the onset of the next saccade, the next fixation point is predicted by analyzing the length of the last saccade distance and / or the last saccade duration.

[0106] An alternative step of predicting the next fixation point may be achieved by determining the gaze direction of the identified next glance and predicting the next fixation point based on the next selectable object within the determined gaze direction.

[0107] The term "next optional object" refers to the object in the scanning direction. In addition, based on the second data set, the next optional object is identifiable and selectable. Therefore, identifiable and optional object comprises the identification probability factor greater than predefined identification probability threshold value and the selection probability factor greater than predefined selection probability threshold value.

[0108] From the beginning of current fixation point and next pan, predict next fixation point by analyzing the pan direction.For example, if the initial indication visual direction of next pan is in the northeast direction, and based on the second data set, the object in the northeast direction can be identified and selectable, then next fixation point will be set at the described object place.If can be identified and selectable more than one object in the pan direction based on the second data set, then next fixation point is set on the object closer to the current fixation point.The described object closer to the current fixation point represents " next optional object " in the context of the present invention.

[0109] If none of the objects within the glance direction are identifiable and selectable based on the second data set, i) the predefined identification probability threshold and / or the predefined selection probability threshold can be lowered to the highest assigned identification probability factor and / or the next fixation point is set to the object with the highest assigned selection probability factor of the identified and identified objects to ensure further processing of the computer-implemented method, or ii) the computer-implemented method stops at this step, does not change the focus of the tunable lens, and continues with the next scenario to tune the tunable lens of the optometry device.

[0110] An alternative step of predicting the next fixation point may be achieved by determining the gaze direction of the identified next glance and predicting the next fixation point based on an object within the determined gaze direction, said object comprising the highest selection probability factor.

[0111] Starting from the current fixation point and the start of the next glance, the next fixation point is predicted by analyzing the glance direction. For example, if the start of the next glance indicates that the visual direction is in the northeast direction, and based on the second data set, multiple objects in the northeast direction are identifiable and selectable, then the next fixation point will be set at the object with the highest selection probability factor of the multiple objects included in the fixation direction.

[0112] If none of the objects within the glance direction are identifiable and selectable based on the second data set, i) the predefined identification probability threshold and / or the predefined selection probability threshold can be lowered to the highest assigned identification probability factor and / or the highest assigned selection probability factor of the identified and identified objects to ensure further processing of the computer-implemented method, or ii) the computer-implemented method stops at this step, does not change the focus of the tunable lens, and continues with the next scenario to tune the tunable lens of the optometry device.

[0113] An alternative step of predicting the next gaze point may be achieved by using a predefined next scene of the second database and assigning a distance of the object to the user of the optometry device based on the first data set and adjusting the estimate (or prediction) of the next gaze point based on the predefined next scene.

[0114] The term "predefined next scene" describes an item of the second database. The item is configured to train a machine learning model or to provide a test scene for a user of the optometry device. Such a test scene is displayed on a screen (e.g., a monitor or a mobile device display). Before the "predefined next scene" is displayed to the user of the optometry device, the scene is evaluated by the computer-implemented method. Before the user of the optometry device identifies the predefined next scene, an identification probability factor and a selection probability factor of an object within the scene are determined. Therefore, before the user of the optometry device identifies the predefined next scene, the tunable lens of the optometry device is tuned based on the scene.

[0115] An alternative step of predicting the next fixation point may be achieved by estimating the next scene by generating a fourth data set comprising EEG activity information of a user of the optometry device; and adjusting the estimate (or prediction) of the next fixation point based on the estimated next scene.

[0116] In particular, in view of the present aspect, the term "fourth data set" refers to the output of the sensor including brain electrical activity information (such as data of an electroencephalogram method or an electrooculogram method). The output can be represented by voltage signals from different areas of brain activity, including information about the amplitude and frequency of the voltage signals, and providing information about the user's next visual and next gaze point.

[0117] The starting point for estimating the next scene includes detecting the current gaze of the user of the optometry device, for example, the right side of the user's visual view. If the user decides to switch his gaze to the left side of the visual view, a brain activity signal is sent to the user's eye. The brain activity signal is detected and analyzed by the sensor for generating a fourth data set.

[0118] Furthermore, a tunable lens of the optometry device is tuned based on the detected and analyzed brain activity signal of the fourth data set.Thereby, the user's gaze is behind the tuning step of the tunable lens.

[0119] An alternative step of predicting the next fixation point may be achieved by estimating the next scene by generating a fifth data set comprising inertial measurement information of the optometry device; and adjusting the estimate (or prediction) of the next fixation point based on the estimated next scene.

[0120] In particular in view of the present aspect, the term "fifth data set" refers to the output of the inertial measurement unit, which includes inertial measurement information (such as angular rate and orientation of the optometric device obtained by using a combination of accelerometers, gyroscopes, and perhaps magnetometers), and provides information about the user's next gaze and next gaze point.

[0121] The starting point for estimating the next scene includes detecting the current vision of the user of the optometry device and inertial measurement information associated with the current vision of the user. If the user decides to switch his vision from one side to the other, a change in the fifth data set is generated. The change in the fifth data set is detected and analyzed by the inertial measurement unit used to generate the fifth data set.

[0122] Furthermore, a tunable lens of the optometry device is tuned based on the detected and analyzed inertial measurement information of the fifth data set.Therefore, the gaze point of the user is after the tuning step of the tunable lens.

[0123] The first aspect of the present invention is entirely solved by a computer-implemented method of the type described above.

[0124] In a second aspect, the present invention relates to a system comprising:

[0125] - Optometry equipment,

[0126] a first device configured to generate a first data set comprising a distance from a user of the optometry device to the object,

[0127] a control unit configured to tune the optometry device according to a distance of a user of the optometry device to the object based on the first data set,

[0128] The system is characterized in that the first device is a LiDAR sensor.

[0129] As generally used, the term "system" refers to at least one device or multiple devices (particularly connected via a computer network). The multiple devices can be connected via a network by using at least one connection interface at any one of the multiple devices.

[0130] The term "control unit" refers to a device that is configured to adjust an optometry device based on a data set generated by other devices. An example of a "control unit" is the Optotune device driver controlled by a PC or Raspberry Pi.

[0131] In a particularly preferred embodiment, the system comprises a second device, wherein the second device is a camera.

[0132] In a particularly preferred embodiment, the system comprises a second device, wherein the second device is an RGB camera.

[0133] In a particularly preferred embodiment of the system, the control unit is configured to tune the optometry device based on a subject identified in the second data set recorded by the second device.

[0134] In a particularly preferred embodiment, the system comprises a third device, said third device being an eye tracker device configured to predict the next gaze of the user based on the third data set.

[0135] In a particularly preferred embodiment of the system, the control unit is configured to select the object based on the identification probability factor and the selection probability factor determined as described above with reference to the computer implemented method.

[0136] The second aspect of the invention is fully solved by a system of the type described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Further features, characteristics and advantages of the present invention will become apparent from the following description of exemplary embodiments of the present invention in conjunction with the accompanying drawings.

[0138] Figure 1 A first exemplary embodiment shows a top view of a user of an optometry device with a tunable lens.

[0139] Figure 2 A block diagram of a first exemplary embodiment of a method for operating a tunable lens by using a LiDAR sensor and an RGB camera is shown.

[0140] Figure 3 A block diagram of a second exemplary embodiment of a method for operating a tunable lens by using a LiDAR sensor, an RGB camera, and an eye tracking device is shown.

[0141] FIG. 4 shows various scenarios to illustrate a first exemplary embodiment of the prediction process.

[0142] FIG. 5 shows various scenarios to illustrate a second exemplary embodiment of the prediction process.

[0143] FIG. 6 shows various scenarios to illustrate a third exemplary embodiment of a prediction process.

[0144] Figure 7 A cross-sectional view of a second exemplary embodiment of an optometry device having a tunable lens is shown.

[0145] exist Figure 1 A first exemplary embodiment of a system 100 for operating a tunable lens 101 is shown in FIG. The system 100 is worn by a user 110 and includes a tunable lens 101, a LiDAR sensor 103, an RGB camera 105, an eye tracking device 117, and a control unit ( Figure 1 not shown).

[0146] like Figure 1 As shown, the LiDAR sensor 103 and the RGB camera 105 simultaneously record a scene 107 observed by the user 110. The scene 107 may include several objects 109, 111, and 115. The data set recorded by the LiDAR sensor 103 includes data representing the respective distances of the user 110 to each of the objects 109, 111, and 115. The data set recorded by the RGB camera 105 includes data representing an image of each of the objects 109, 111, and 115. The eye tracking device 117 records the movement of the eyes of the user 110 based on the observed scene 107.

[0147] Based on the second data set recorded by the RGB camera 105, the object 107 and the object 109 are recognized, identified and selected by the control unit of the system 100. In addition, based on the first data set recorded by the LiDAR sensor 103, the respective distances of the user 110 to each of the objects 107 and 109 are determined by the control unit of the system 100. The control unit of the system 100 assigns each distance to the respective object 109, 111 and 115 in a next step. Based on the selection of the control unit and the determined distances of the selected objects, the tunable lens 101 is tuned according to the data of the LiDAR sensor and the data of the RGB camera.

[0148] Optionally, the data of the eye tracking device 117 may be processed by a control unit of the system 100 and used to tune the tunable lens 101 based on the data of the LiDAR sensor, the data of the RGB camera and the data of the eye tracking device.

[0149] Figure 2 The process steps of a first exemplary embodiment of a method 200 for operating a tunable lens 101 , 701 by using the LiDAR sensor 103 , 703 and the RGB camera 105 , 705 of the systems 100 and 700 are shown.

[0150] In the first data generation step 201 of the first exemplary embodiment of the method 200, a first data set 211 and a second data set 212 are generated. The first data set 211 is generated by a LiDAR sensor, and the second data set 212 is generated by an RGB camera. The first data set 211 and the second data set 212 both include the same scene 107 (see Figure 1 ), wherein the scene includes object 109, object 111 and object 115.

[0151] Based on second data set 212 , identification step 203 is performed. In a first sub-step of identification step 203 , an identification probability factor 283 is determined for each of objects 109 , 111 , and 115 of scene 107 based on second data set 212 and first database 281 .

[0152] In an exemplary embodiment, object 109 may include an identification probability factor 283 of 0.8, object 111 may include an identification probability factor 283 of 0.7, and object 115 may include an identification probability factor 283 of 0.6.

[0153] In a second sub-step of the identification step 203, the identification probability factor 283 of each identified object 109, 111 and 115 is compared with a predefined identification probability threshold 285. At the end of said second sub-step of the identification step 203, objects having an identification probability factor 283 below the predefined identification probability threshold 285 are ignored for further steps. In the above exemplary embodiment, the predefined identification probability threshold 285 may include a value of 0.7. Thus, for further steps, objects 109 and 111 including an identification probability factor 283 equal to or higher than 0.7 will be considered, and for further steps, object 115 including an identification probability factor 283 of 0.6 will not be considered.

[0154] Based on the second data set 212 and the identification step 203 , a selection step 205 is performed.

[0155] In a first sub-step of the selection step 205, the remaining objects 109 and 111 of the scene 107 having identification probability factors equal to or greater than the identification probability threshold are assigned a selection probability factor 293 based on the second data set 212 and the second database 291. In the above exemplary embodiment, the object 109 may include a selection probability factor 293 of 0.8, and the object 111 may include a selection probability factor 293 of 0.7.

[0156] In a second sub-step of the selection step 205, the selection probability factor 293 of each identified object 109 and 111 is compared to a predefined selection probability threshold 295. At the end of this second sub-step of the selection step 205, objects having a selection probability factor 293 below the predefined selection probability threshold 295 are ignored for further steps. If more than one object exceeds the predefined selection probability threshold 295, the object 297 comprising the higher or highest selection probability factor 293 is selected. In the above-described exemplary embodiment, the predefined selection probability threshold 295 may comprise a value of 0.7. Therefore, for further steps, objects 109 and 111 comprising a selection probability factor 293 equal to or higher than 0.7 may be considered. The selection probability factor 293 of object 109 is higher than the selection probability factor 293 of object 111. Therefore, for further steps, object 109 will be considered.

[0157] Based on the first data set 211 and the selected object 297 , an assigning step 207 is performed. In the assigning step 207 , based on the first data set 211 , a distance of the selected object 297 is assigned to the selected object 297 .

[0158] Based on and after the assigning step 207, a tuning step 209 is performed. In the tuning step 209, the tunable lens 101 is tuned based on the selected object 297 and the distance of the selected object 297 based on the first data set 211. This results in a tuned tunable lens 101.

[0159] Figure 3 A second exemplary embodiment of a method 300 for operating a tunable lens by using a LiDAR sensor 103 , an RGB camera 105 , and an eye tracking device 117 is shown.

[0160] In the data generation step 301 of the second exemplary embodiment of the method 300, a first data set 311, a second data set 312, and a third data set 313 are generated. The first data set 311 is generated by the LiDAR sensor 103, the second data set 312 is generated by the RGB camera 105, and the third data set 313 is generated by the eye tracking device 117. The first data set 311 and the second data set 312 both include the same scene 107, wherein the scene includes the object 109, the object 111, and the object 115 (see Figure 1 ). The third data set 313 includes the glance information and gaze point information of the user 110 in the context of the scene 107. In other words, the first data set 311 and the second data set 312 represent the scene 107, while the third data set 313 represents the movement of the user's eyes triggered by the scene 107.

[0161] Based on the third data set 313, a determination step 315 is performed. In the determination step 315, a current gaze point 361 of the user 110 of the tunable lens 101 is obtained based on the third data set 313. In an exemplary embodiment, the current gaze point 361 is set to the object 109 on the left side of the identified scene 207.

[0162] Based on the current gaze point 361 and the third data set 313, a change in the gaze position 362 is detected by the tracking step 317. The change in the gaze position 363 may result in a new gaze point. In the above exemplary embodiment, the gaze position changes from the left side of the identified scene 107 to the right side of the identified scene 107.

[0163] Based on the changes in gaze position 363 detected by tracking step 317, an estimated gaze point 365 is estimated by prediction step 319. In the exemplary embodiment, estimated gaze point 365 is set to object 107 on the right side of identified scene 107.

[0164] Based on the second data set 312, an identification step 303 similar to the identification step 203 of the previous first exemplary embodiment for operating a tunable lens and a selection step 305 similar to the selection step 205 of the previous first exemplary embodiment for operating a tunable lens are performed. At the end of the selection step 305, one or more selectable objects 397 including a selection probability factor greater than a predefined selection probability threshold may be suitable for setting a next fixation point.

[0165] Based on the first data set 311, the selectable objects 397 based on the second data set 312 and the estimated fixation point 365, an assignment step 307 is performed. In said assignment step 307, the estimated distance of the fixation object 107 to the user 110 is assigned to the new fixation point 365, the distance being based on the first data set 311 and the estimated position of the fixation object 107. In other words, the computer-implemented method generates in the assignment step 307 information setting the estimated fixation point 365 to the object 107 if the object 107 has a selection probability factor greater than a predefined selection probability threshold.

[0166] Based on the assigning step 307, a tuning step 309 is performed. In said tuning step 309, the tunable lens 101 is tuned based on the estimated fixation object 107 and the estimated distance of the fixation object 107 based on the first data set 311. This results in a tuned tunable lens 101.

[0167] With the aid of FIG4 , a first exemplary embodiment of a prediction process 400 based on this method is explained in more detail. FIG4 is subdivided into Figure 4a )to Figure 4c). Each segmentation graph represents the same scene observed by the user at different times t1 to t3. This first exemplary embodiment is based on determining the identified gaze direction of the next glance and on predicting the next gaze point based on the next selectable object in the determined gaze direction.

[0168] Figure 4a ) shows a scene 407 including an object 409, an object 411, and an object 415, and represents the start of the prediction process at time t1=0s. Assume that the current fixation point 461 is set at the object 409 at the time t1=0s. The current fixation point 461 is represented as a circle, which includes all possible micro-saccades within the fixation point. As long as the current fixation point 461 at least partially covers the object 409, the current fixation point 461 will remain set at the object 409.

[0169] Figure 4b ) shows a scene 407 including objects 409, 411, and 415, and represents a current glance 463 in the prediction process at time t2=0.5s. The current glance 463 is along the direction d1 (see Figure 4b ) in the figure. The current glance 463 will keep moving until the user of the tunable lens looks at a new object.

[0170] Based on the detected glance, the computer-implemented method predicts the user's new gaze point 465 by determining the direction d1 of the current glance 463 and detecting the next object within the direction d1 of the current glance 463, wherein the next object includes i) an identification probability factor not lower than a predefined identification probability threshold and ii) a selection probability factor not lower than a predefined selection probability threshold.

[0171] In the numerical example of the exemplary first embodiment of the prediction process 400, it is assumed that the predefined identification probability threshold is set to 0.7, and it is assumed that the predefined selection probability threshold is set to 0.7. If an object 415 in the direction d1 of the detected glance includes an identification probability factor of 0.8 and a selection probability factor of 0.8, and another object 411 in the direction d1 of the detected glance includes an identification probability factor of 0.8 and a selection probability factor of 0.9, then both object 415 and object 411 are suitable for setting the predicted fixation point 465.

[0172] Figure 4c) shows a scene 407 including an object 409, an object 411, and an object 415, and represents the end of the prediction process at time t3=1.0s. At the time t3=1.0s, the predicted fixation point 465 is set at the object 415. Since the object 415 is the next optional object in the direction d1 of the pan at time t2, the predicted fixation point 465 is set at the object 415. If the object 415 is not in the direction d1 of the pan at time t2, the predicted fixation point 465 will be set at the object 411.

[0173] With the aid of FIG5 , a second exemplary embodiment of a prediction process 500 based on this method is explained in more detail. FIG5 is subdivided into Figure 5a )to Figure 5c ). Each segmented graph represents the same scene observed by the user at different times t1 to t3. This second exemplary embodiment is based on determining the identified gaze direction of the next glance and on predicting the next gaze point based on an object within the determined gaze direction, the object comprising the highest selection probability factor.

[0174] Figure 5a ) shows a scene 507 including an object 509, an object 511, and an object 515, and represents the start of the prediction process at time t1=0s. Assume that the current fixation point 561 is set at the object 509 at the time t1=0s. The current fixation point 561 is represented as a circle, which includes all possible micro-saccades within the fixation point. As long as the current fixation point 561 at least partially covers the object 509, the current fixation point 561 will remain set at the object 509.

[0175] Figure 5b ) shows a scene 507 including objects 509, 511, and 515, and represents a current glance 563 in the prediction process at time t2=0.5s. At said time t2=0.5s, the current glance 563 moves along the direction d1 (see Figure 5b ). The current glance 563 will keep moving until the user of the tunable lens looks at a new object.

[0176] Based on the detected glance, the computer-implemented method predicts the user's new gaze point 565 by determining the direction d1 of the current glance 563 and detecting objects within the direction d1 of the current glance 563, which objects include i) an identification probability factor not lower than a predefined identification probability threshold and ii) a selection probability factor not lower than a predefined selection probability threshold, and in the case of multiple objects, the object with the highest selection probability.

[0177] As with the example described with reference to FIG4 , assume that the predefined identification probability threshold is set to 0.7, and assume that the predefined selection probability threshold is set to 0.7. If object 515 includes an identification probability factor of 0.8 and a selection probability factor of 0.8, and object 511 includes an identification probability factor of 0.8 and a selection probability factor of 0.9, then both object 515 and object 511 are suitable for setting predicted fixation point 565.

[0178] Figure 5c ) shows a scene 507 including an object 509, an object 511, and an object 515, and represents the end of the prediction process at time t3=1.0s. At the time t3=1.0s, the predicted fixation point 565 is set at the object 511. Since the object 511 is an optional object in the direction d1 of the pan at time t2, the optional object includes the highest selection probability factor, so the predicted fixation point 565 is set at the object 511. If the object 511 is not in the direction d1 of the pan at time t2, the predicted fixation point 565 will be set at the object 515.

[0179] With the aid of FIG6 , a third exemplary embodiment of a prediction process 600 based on this method is explained in more detail. FIG6 is subdivided into Figure 6a )to Figure 6c ). Each segmented graph represents the same scene observed by the user at different times t1 to t3. This third exemplary embodiment is based on determining the gaze direction of the next identified glance and on predicting the next gaze point based on the user's previous gaze information.

[0180] Figure 6a ) shows a scene 607 including an object 609, an object 611, and an object 615, and represents the start of the prediction process at time t1=0s. Assume that the current fixation point 661 is set at the object 609 at the time t1=0s. The current fixation point 661 is represented as a circle, which includes all possible micro-saccades within the fixation point. As long as the current fixation point 661 at least partially covers the object 609, the current fixation point 661 will remain set at the object 609.

[0181] Figure 6b ) shows a scene 607 including an object 609, an object 611 and an object 615, and represents a current glance 663 in the prediction process at time t2=0.5s. At said time t2=0.5s, the current glance 663 moves along the direction d1. The current glance 663 will keep moving until the user of the tunable lens is looking at a new object. Based on the user's previous visual information, the estimated length of the current glance 663 is estimated by the vector d2.

[0182] Based on the detected glance, the computer-implemented method predicts the user's gaze point 665 by determining the direction d1 of the current glance 663 and estimating the length of the current glance 663 by using the length of the vector d2. Adding the vector d2 to the gaze point 661 at time t1=0s can result in the predicted gaze point 665. If i) the predicted gaze point 665 at least partially covers the object in the direction d1, and ii) if the object includes an identification probability factor that is not lower than a predefined identification probability threshold, and iii) if the object includes a selection probability factor that is not lower than a predefined selection probability threshold, the predicted gaze point 665 will be set.

[0183] As in the example described with reference to FIGS. 4 and 5 in the third embodiment of the prediction process 600, it is assumed that the predefined identification probability threshold is set to 0.7, and it is assumed that the predefined selection probability threshold is set to 0.7. If it is assumed that object 615 includes an identification probability factor of 0.8 and a selection probability factor of 0.8, and it is assumed that object 611 includes an identification probability factor of 0.8 and a selection probability factor of 0.9, then both object 615 and object 611 are suitable for setting the predicted fixation point 665.

[0184] Figure 6c ) shows a scene 607 including an object 609, an object 611 and an object 615, and represents the end of the prediction process when time t3=1.0s. When the time t3=1.0s, based on the vector d2, the predicted fixation point 665 is set at the object 615. Since the object 615 is covered by the predicted fixation 665 based on the vector d2 and is located in the direction d1 of the pan at the time t2, the predicted fixation point 665 is set at the object 615. Since the distance of the object 665 from the actual fixation point 609 does not meet the length and direction of the vector d2, the object 611 will not be selected for setting the predicted fixation point 665. Therefore, the object 611 is not covered by the predicted fixation point 665, although the object 611 is located in the direction d1 of the pan at the time t2 and includes a higher selection probability factor (compared with the object 615).

[0185] Figure 7 An optometry device 700 with a tunable lens 701 is shown, which can be operated with the method according to the present invention. The optometry device 700 is configured to be worn by a user 710 and comprises a first tunable lens 701a for the user's right eye R and a second tunable lens 701b for the user's left eye L. Both tunable lenses 701a, 701b are arranged in a frame 731 configured to be worn by the user 710. In an alternative embodiment, the frame 731 can be part of the housing of the optometry device 700 configured to determine the optometry prescription value of the user 710.

[0186] The frame 731 includes a LiDAR sensor 703, an RGB camera 705, and an eye tracking device or two eye tracking devices 717a, 717b, one for detecting the movement of the right eye R of the user 710, and one for detecting the movement of the left eye L of the user 710. The viewing direction of the LiDAR sensor 703 and the camera 705 is directed to the viewing direction of the right eye R and the left eye L of the user 710. The optics of the LiDAR sensor 703 and the RGB camera 705 are configured so that the two devices record data of almost the same scene. The detection direction of the eye tracking devices 717a, 717b is almost in the opposite direction of the viewing direction of the LiDAR sensor 703 and the camera 705, that is, in the direction of the eyes R, L of the user 710.

[0187] The optometry device 700 further comprises a control unit 730 which receives data from the LiDAR sensor 703, the RGB camera 705 and the eye tracking devices 717a, 717b. The control unit 730 generates control data for the tunable lenses 701a, 701b, as described in more detail above. In addition, the control unit 730 generates output data describing the correction values ​​required to correct the viewing ability of the eyes R, L of the user 710, which can be output to and displayed on a display or any other suitable output device 732. The control unit 730 can also be arranged on the frame 731, but can also be arranged separately from the frame 731, for example in a console for operating the optometry device.

Claims

1. A computer-implemented method (200, 300) for operating an optometry device (700), the method comprising the following steps: - generating (201, 301) a first data set (211, 311), said first data set (211, 311) comprising information about a distance from a user (110, 710) of the optometry device (700) to an object (109, 111, 115), - tuning (209, 309) a tunable lens (101, 701) of the optometry device (700) according to a distance from a user (110, 710) of the optometry device (700) to the object (109, 111, 115) based on the first data set (211, 311), the first data set (211, 311) is generated by using a LiDAR sensor (103, 703) which measures the distance from the user (110, 710) of the optometry device (700) to the object (109, 111, 115) by evaluating the scene (107, 407, 507, 607) represented by the first data set (211, 311), characterized in that - generating (201, 301) a second data set (212, 312) by recording an image of the scene (107, 407, 507, 607) by a camera, and - the further step of identifying (203, 303) the object (109, 111, 115) by evaluating the scene (107, 407, 507, 607) represented by the second data set (212, 312).

2. The computer-implemented method (200, 300) of claim 1, wherein: The recording of the image is performed by using an RGB camera (105, 705).

3. The computer-implemented method (200, 300) of claim 1, characterized in that: Identification (203, 303) This object includes: - identifying a plurality of objects (109, 111, 115) in an image of the scene (107, 407, 507, 607) represented by the second data set (212, 312), and - generating an identification probability factor (283) for each identified object (109, 111, 115) of the plurality of objects (109, 111, 115) using the first database (281), and - assigning the identification probability factor (283) to each object (109, 111, 115) of the plurality of objects (109, 111, 115) based on the first database (281), and - comparing the identification probability factor (283) of each identified object (109, 111, 115) of the plurality of objects (109, 111, 115) with a predefined identification probability threshold (285), and - generating a selection probability factor (293) using a second database (291) for each identified object (287, 387) of the plurality of objects (109, 111, 115) having an identification probability factor (283) equal to or greater than the predefined identification probability threshold (285), and - comparing the selection probability factor (293) of at least each identified object (287, 387) of the plurality of objects having an identification probability factor (283) equal to or greater than the predefined identification probability threshold (285) with the predefined selection probability threshold (295), and - determining a distance from a user (110, 710) of the optometry device (700) to an identified object (287, 387) having a maximum selection probability factor (293), wherein the selection probability factor (293) of the identified object (287, 387) is greater than the predefined selection probability threshold (295).

4. The computer-implemented method (200, 300) of claim 3, characterized in that: A machine learning algorithm is used to determine the identification probability factor (283) stored in the first database (281) and the selection probability factor (293) stored in the second database (291).

5. The computer-implemented method (200, 300) according to any one of claims 1 to 4, characterized in that: A third data set (303) is generated (201, 301), the third data set (303) comprising visual information of a user (110, 710) of the optometry device (700), a next visual observation of the user (110, 710) is predicted (400, 500, 600) based on the third data set (303), and a tunable lens (101, 701) of the optometry device (700) is tuned (209, 309) based on the predicted next visual observation of the user (110, 710).

6. The computer-implemented method (200, 300) of claim 5, characterized in that: Predicting (400, 500, 600) the next visual look of the user (110, 710) includes the following steps: - determining (315) a current gaze point (361, 461, 561, 661) of a user (110, 710) of the optometry device (700) based on the third data set (303), and - assigning the object based on the second data set (212, 312) and the distance of the object to the user (110, 710) of the optometry device (700) based on the first data set (211, 311) to the current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700), and - tracking (317) a current gaze point (361, 461, 561, 616) of a user (110, 710) of the optometry device (700) based on the third data set (303) until a next glance (363, 463, 563, 663) of the user (110, 710) of the optometry device (700) is identified by detecting a change in gaze position, and - predicting (319) a next gaze point (365, 465, 565, 665) of the user (110, 710) of the optometry device (700) based on a current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700) and an identified next glance (363, 463, 563, 663) of the user (110, 710) of the optometry device (700), and - tuning (209, 309) the optometry device (700) according to the predicted next fixation point (365, 465, 565, 665).

7. The computer-implemented method (200, 300) of claim 6, characterized in that: Predicting (400, 500, 600) the next fixation point (365, 465, 565, 665) comprises at least one of the following steps: - determining the gaze direction (d1) of the identified next glance (363, 463, 563, 663) and predicting the next gaze point (365, 465, 565, 665) based on the previous gaze information (d2) of the user (110, 710) - determining a gaze direction (d1) of the identified next glance (363, 463, 563, 663) and predicting the next gaze point (365, 465, 565, 665) based on the next selectable object within the determined gaze direction (d1) -Determining the gaze direction (d1) of the identified next glance (363, 463, 563, 663), and predicting the next gaze point (365, 465, 565, 665) based on an object (109, 111, 115) within the determined gaze direction (d1), the object (109, 111, 115) including the highest selection probability factor (293). - using a predefined next scene of the second database (291) and assigning a distance of the object to a user (110, 710) of the optometry device (700) based on the first data set (211, 311) and adjusting an estimate of the next gaze point (365, 465, 565, 665) based on the predefined next scene; - estimating a next scene by generating a fourth data set, the fourth data set comprising brain electrical activity information of a user (110, 710) of the optometry device (700), and adjusting an estimation of the next fixation point (365, 465, 565, 665) based on the estimated next scene; - estimating a next scene by generating a fifth data set comprising inertial measurement information of the optometry device (110, 710), and adjusting an estimate of the next fixation point (365, 465, 565, 665) based on the estimated next scene.

8. A computer-implemented method (200, 300) for operating an optometry device (700), the method comprising the steps of: - generating (201, 301) a first data set (211, 311), said first data set (211, 311) comprising information about a distance from a user (110, 710) of the optometry device (700) to an object (109, 111, 115), - tuning (209, 309) a tunable lens (101, 701) of the optometry device (700) according to a distance from a user (110, 710) of the optometry device (700) to the object (109, 111, 115) based on the first data set (211, 311), the first data set (211, 311) is generated by using a LiDAR sensor (103, 703) which measures the distance from the user (110, 710) of the optometry device (700) to the object (109, 111, 115) by evaluating the scene (107, 407, 507, 607) represented by the first data set (211, 311), characterized in that - generating (201, 301) a third data set (303), the third data set (303) comprising visual information of a user (110, 710) of the optometry device (700), predicting (400, 500, 600) the next visual observation of the user (110, 710) based on the third data set (303), and tuning (209, 309) a tunable lens (101, 701) of the optometry device (700) based on the predicted next visual observation of the user (110, 710).

9. The computer-implemented method (200, 300) of claim 8, characterized in that: Predicting (400, 500, 600) the next visual look of the user (110, 710) includes the following steps: - determining (315) a current gaze point (361, 461, 561, 661) of a user (110, 710) of the optometry device (700) based on the third data set (303), and - assigning the object based on the second data set (212, 312) and the distance of the object to the user (110, 710) of the optometry device (700) based on the first data set (211, 311) to the current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700), and - tracking (317) a current gaze point (361, 461, 561, 616) of a user (110, 710) of the optometry device (700) based on the third data set (303) until a next glance (363, 463, 563, 663) of the user (110, 710) of the optometry device (700) is identified by detecting a change in gaze position, and - predicting (319) a next gaze point (365, 465, 565, 665) of the user (110, 710) of the optometry device (700) based on a current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700) and an identified next glance (363, 463, 563, 663) of the user (110, 710) of the optometry device (700), and - tuning (209, 309) the optometry device (700) according to the predicted next fixation point (365, 465, 565, 665).

10. The computer-implemented method (200, 300) of claim 9, characterized in that: Predicting (400, 500, 600) the next fixation point (365, 465, 565, 665) comprises at least one of the following steps: - determining the gaze direction (d1) of the identified next glance (363, 463, 563, 663) and predicting the next gaze point (365, 465, 565, 665) based on the previous gaze information (d2) of the user (110, 710) - determining a gaze direction (d1) of the identified next glance (363, 463, 563, 663) and predicting the next gaze point (365, 465, 565, 665) based on the next selectable object within the determined gaze direction (d1) -Determining the gaze direction (d1) of the identified next glance (363, 463, 563, 663), and predicting the next gaze point (365, 465, 565, 665) based on an object (109, 111, 115) within the determined gaze direction (d1), the object (109, 111, 115) including the highest selection probability factor (293). - using a predefined next scene of the second database (291) and assigning a distance of the object to a user (110, 710) of the optometry device (700) based on the first data set (211, 311) and adjusting an estimate of the next gaze point (365, 465, 565, 665) based on the predefined next scene; - estimating a next scene by generating a fourth data set, the fourth data set comprising brain electrical activity information of a user (110, 710) of the optometry device (700), and adjusting an estimation of the next fixation point (365, 465, 565, 665) based on the estimated next scene; - estimating a next scene by generating a fifth data set comprising inertial measurement information of the optometry device (110, 710), and adjusting an estimate of the next fixation point (365, 465, 565, 665) based on the estimated next scene.

11. A system (100), comprising: - Optometry equipment (700), a first device configured to generate a first data set (211, 311), the first data set (211, 311) comprising a distance from a user (110, 710) of the optometry device (700) to an object (109, 111, 115), a control unit (730) configured to tune the optometry device (700) according to a distance of the user (110, 710) of the optometry device (700) to the object based on the first data set (211, 311), - The first device is a LiDAR sensor (103, 703) It is characterized in that - further comprising a second device, wherein the second device is a camera, and The control unit (730) is configured to tune the optometry device (700) based on the object (109, 111, 115) identified in a second data set (212, 312) recorded by the second device.

12. The system according to claim 11, characterized in that The camera is an RGB camera (105, 705).

13. The system according to any one of claims 11 to 12, characterized in that Further included is a third device, the third device being an eye tracker device (117, 717a, 717b) configured to predict a next gaze of the user (110, 710) based on the third data set (303).

14. The system according to any one of claims 11 to 13, characterized in that The control unit (730) is configured to select the object (109, 111, 115) based on the identification probability factor (283) and the selection probability factor (293).

15. A system (100), comprising: - Optometry equipment (700), a first device configured to generate a first data set (211, 311), the first data set (211, 311) comprising a distance from a user (110, 710) of the optometry device (700) to an object (109, 111, 115), a control unit (730) configured to tune the optometry device (700) according to a distance of the user (110, 710) of the optometry device (700) to the object based on the first data set (211, 311), -The first device is a LiDAR sensor (103, 703). It is characterized in that - further comprising a second device, wherein the second device is a camera, and - further comprising a third device, the third device being an eye tracker device (117, 717a, 717b) configured to predict a next gaze of the user (110, 710) based on the third data set (303).

16. The system according to claim 15, characterized in that The camera is an RGB camera (105, 705).

17. The system according to any one of claims 15 to 16, characterized in that The control unit (730) is configured to select the object (109, 111, 115) based on the identification probability factor (283) and the selection probability factor (293).

Images