Method and system for operating an optometric device

Abstract

The invention relates to a computer-implemented method for operating an optometry device, the method comprising the steps of: - generating a first data set comprising information about a distance of a user of the optometry device to an object; - tuning a tunable lens of the optometry device according to the distance of the user of the optometry device to the object based on the first data set. The computer-implemented method of the invention is characterized in that the first data set is generated by using a LiDAR sensor, the LiDAR sensor measuring the distance of the user of the optometry device to the object by evaluating a scene represented by the first data set.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for operating an optometric device as described in the preamble of claim 1, and a system comprising the optometric device as described in the preamble of claim 10. Background Technology

[0002] US 2022 / 0146819 A1 discloses an apparatus and method for visual depth prediction. The apparatus includes a camera assembly and a controller. The camera assembly is configured to capture images of a user's two eyes. Using the captured images, the controller determines the position of each pupil in each of the user's eyes. The determined pupil positions and the captured images are used to determine eye-tracking parameters, which are used to calculate the value of an eye-tracking function. Using the calculated values ​​and a model that maps the eye-tracking function to visual depth, the user's visual depth is determined.

[0003] EP 3 498 149 A1 discloses an optoelectronic binocular instrument for automatically correcting presbyopia and a method for binocular correction of presbyopia. The instrument has two optoelectronic lenses and a capture subsystem for capturing images of the eyes. By means of pupil tracking that processes the images of the eyes, the system determines the distance at which the subject is viewing. The pupil tracking operates at very high speed using a high-performance graphics processor and a highly parallelized algorithm for pupil tracking. The method consists of two stages. In the first stage, calibration is performed by requiring the subject to view a target at different distances and measuring the size and position of the pupils. In the second stage, correction is performed by the instrument; the system continuously captures and processes images to calculate the correction to be applied, and finally corrects the presbyopia by applying said correction.

[0004] US 7,656,509 B2 discloses devices for determining distance. These devices present an electrically activated lens that the user is looking at. Once the distance is determined, the devices can adjust the optical power of the electrically activated lens to ensure the object is properly focused. Optical rangefinding is a possible means of performing this task. An active rangefinder can emit light radiation directed at the object from a transmitter. The light radiation can then be reflected by the object. The reflected light radiation can then be received by a suitable receiver. The received light radiation can then be processed by a suitable circuit system to determine the distance to the object. Passive rangefinders do not require a transmitter. Instead, a suitable receiver receives ambient light from the object. The received light can then be processed by a suitable circuit system to determine the distance to the object.

[0005] US Patent 11,221,488 B1 discloses a pair of eyeglasses comprising one or more adjustable lenses, each configured to align with a user's corresponding eye. These adjustable lenses include a concave liquid crystal adjustable lens stacked with non-liquid crystal adjustable lenses (e.g., fluid-filled lenses or Alvarez lenses). The concave adjustable lens comprises an electrically modulated optical material, such as one or more liquid crystal cells. Each liquid crystal cell includes an array of electrodes extending in one, two, three, four, or more directions. A control circuitry system can apply control signals to the electrode array in each liquid crystal cell to produce a desired phase profile. Each lens can be centrally concave such that the portion of the lens within the user's field of vision exhibits a different phase profile than the portion outside the user's field of vision.

[0006] US2020 / 379214 A1 discloses an augmented reality (AR) device including a zoom lens whose focal length can be changed by adjusting its refractive power and by adjusting the position of a focus adjustment area of ​​the zoom lens according to the user's viewing direction. The AR device can use an eye tracker to obtain an eye vector indicating the user's viewing direction, adjust the refractive power of a first focus adjustment area of ​​the first zoom lens to change the focal length used to display a virtual image, and complementaryly adjust the refractive power of a second focus adjustment lens relative to the adjusted refractive power of the first focus adjustment area.

[0007] US2018 / 081041 A1 relates to a lidar depth sensing device. The device includes: a laser that emits light radiation pulses toward a scene; and one or more detectors that receive light radiation reflected from points in the scene.

[0008] Hasan, N., Karkhanis, M., Khan, F., Ghosh, T., Kim, H., and Mastrangelo, CH. (2017), in *Adaptive Optics for Autofocusing Eyeglasses*, published in *Optics InfoBase Conference Papers* (https: / / doi.org / 10.1364 / AIO.2017.AM3A.1), describes an implementation of adaptive glasses designed to restore accommodative ability lost due to age-related presbyopia. The adaptive variable refractive power eyepiece is driven by a battery-powered microcontroller system that measures the distance from the observer to the object and combines it with the observer's optometric prescription to produce a clear image at any object distance.

[0009] The paper "Portable device for presbyopia correction with optoelectronic lenses driven by pupil response" by Mompeán, J., Aragón, JL, and Artal, P. (2020), published in Scientific Reports, 10(1) (https: / / doi.org / 10.1038 / s41598-020-77465-5), describes a portable device developed and constructed to dynamically and automatically correct presbyopia using a pair of optoelectronic lenses driven by pupil tracking. The system is entirely portable and provides a high defocus correction range of up to 10D. The glasses are controlled and powered by a smartphone. To achieve a truly real-time response, an image processing algorithm has been implemented in OpenCL and run on the smartphone's GPU. To validate the system, various visual experiments were conducted on presbyopic subjects. Visual acuity remained almost constant across distances ranging from 5m to 20cm.

[0010] Padmanaban, N., Konrad, RK, and Wetzstein, G. (2019), in their paper "Autofocals: Evaluating gaze-contingent eyeglasses for presbyopes," included in ACM SIGGRAPH 2019 Talks, SIGGRAPH 2019, Association for Computing Machinery, Inc. (https: / / doi.org / 10.1145 / 3306307.3328147), describes the estimation of the depth of the gazed object, dynamically estimated via sensor fusion of four "raw" inputs: two gaze-tracking cameras, a scene-oriented depth camera, and the user's interpupillary distance (IPD). A binocular eye tracker estimates the convergence-divergence distance at 120 Hz. However, small errors in gaze direction estimation introduce significant biases into the estimated convergence-divergence distance. Although the depth sensor operates at only 30Hz, it, along with the visual orientation, compensates for biases in convergence and divergence measurements. The authors developed a custom sensor fusion algorithm to balance the accuracy and velocity of convergence and divergence estimation of the pipeline.

[0011] Jarosza J., Molliexa N., Lavignea Q., and Berge B. (2019), in *An original low-power opto-fluidic engine for presbyopia-correcting adaptive eyeglasses*, published in *Ophthalmic Technologies XIX*, edited by Fabrice Manns, Per G. Soderberg, and Arthur Ho, SPIE Proceedings, Vol. 10858, 1085824, doi: 10.1117 / 12.2507816, describes a concept for presbyopia-correcting adaptive eyeglasses for people who do not receive corrective solutions. The proposed eyeglasses automatically provide clear vision at all distances and feature enhanced field of vision and high optical quality. The adaptive technology relies on a primitive fluid-filled lens whose focusing capability is set by a low-power microfluidic pump inserted into the temple. A LiDAR sensor is used to measure the focusing distance.

[0012] Existing tunable lenses typically include an IR camera for scene detection and an eye-tracking device for depth recognition. Existing tunable lenses generally have limitations in depth resolution and distance estimation accuracy. Furthermore, the tuning of known tunable lenses is performed after the user's gaze point has been determined. Therefore, the tuning of the tunable lens lags behind the user's gaze process. Additionally, scene context is typically not considered when operating tunable lenses.

[0013] The problem to be solved

[0014] Therefore, especially considering the publications of US11 221 488 B1 and Jarosza et al., the object of the present invention is to provide a computer-implemented method and system that overcomes the aforementioned limitations of the prior art.

[0015] The specific objective of this invention is to provide a computer-implemented method and system for more accurate and faster distance measurement, enabling more accurate and natural tuning of optometric equipment. Summary of the Invention

[0016] The problem is solved by a computer-implemented method and a system that includes optometry equipment.

[0017] In a first aspect, the present invention relates to a computer-implemented method for operating an optometric device, the method comprising the following steps:

[0018] - Generate a first dataset, which includes information about the distance from the user to the object in the optometry device.

[0019] - The tunable lens of the optometry device is tuned based on the distance from the user to the object of the optometry device, which is based on the first dataset.

[0020] The computer-implemented method is characterized in that the first dataset is generated using a LiDAR sensor, which measures the distance from the user of the optometry device to an object by evaluating the scene represented by the first dataset.

[0021] As commonly used, the term "computer-implemented method" refers to a method involving at least one device (specifically a computer) or multiple devices (particularly connected via a computer network). The multiple devices can be connected via a network using at least one connection interface at any one of the devices. A computer-implemented method can be implemented as at least one computer program available on a storage medium carrying a computer program, thereby performing at least one step of the computer-implemented method by using the at least one computer program. Alternatively, the at least one computer program can be accessed by a device suitable for executing the method via a network (e.g., via an intranet or via the Internet).

[0022] The term "optometry equipment" refers to a device that processes light waves. Within the scope of this invention, the term "optometry equipment" refers to at least spectacle lenses, binoculars, contact lenses, or any other device for correcting or detecting eye problems in a user.

[0023] The term "dataset" in the context of this invention describes an item comprising at least one piece of information, such as a numeric or alphanumeric item. The data may be provided in a machine-readable form, such that it may be the input or output of a machine learning model, particularly the input or output of any neural network included in the machine learning model, or it may be the input or output of a computer-implemented method. Specifically, with regard to this aspect, the term "first dataset" refers to the output of a LiDAR sensor. This output may be represented by point clouds or digital information about the distances to multiple objects in a scene.

[0024] As is commonly used, the term “tuning” or any of its grammatical variations refers to the process of adjusting the optical power of an optometric device.

[0025] The term "optical power" is a collective term for the spherical power (which brings a near-axis parallel beam of light to a single focal point) and the cylindrical power (which brings a near-axis parallel beam of light to two separate focal lines perpendicular to each other (DIN ISO 13666:2019, Section 3.10.2)) of spectacle lenses.

[0026] The term "tunable lens" refers to a light-focusing device (in the form of a fluid, gel, or solid) that can dynamically modulate its focal length using input electrical energy. Therefore, in a tunable lens, some internal properties that alter the shape of the light wavefront can be modified electrically.

[0027] Examples of tunable lenses include shape-changing lenses based on a combination of optical fluid and polymer films. The core element consists of a container filled with an optical fluid and sealed with a thin, elastic polymer film. The optical fluid can be a transparent liquid with specific optical properties, such as refractive index. A ring pushed towards the center of the film shapes the tunable lens. The deflection of the film and the radius of the lens can be changed by pushing the ring towards the film, by applying pressure to the outside of the film, or by pumping the liquid into or out of the container. Other examples of tunable lenses are Alvarez lenses, which shift a pair of complementary cubic phase plates relative to each other to change the optical power; and lenses employing liquid crystal technology, which change the orientation of liquid crystals with the help of an electric current, thereby changing the refraction of light.

[0028] The term "LiDAR" is an acronym for "light detection and ranging" or "laserimaging, detection, and ranging." LiDAR sensors are used to determine distance by aiming a laser at an object or surface and measuring the time it takes for the reflected light to return to a receiver. In this invention, it is used for distance measurement in mobile applications.

[0029] As commonly used, the term "evaluation" or any grammatical variation thereof refers to the process of analyzing a scene. The term "scene" in the context of this invention describes the field of view of a person or a digital device (such as a LiDAR sensor or camera). In the context of this invention, a scene is represented by an image, video, or dataset. As used herein, the term "image"

[0030] This refers to a single image, especially an image of the environment. As used in this article, the term "video"...

[0031] It refers to multiple images of the environment.

[0032] By utilizing this computer-implemented method (which includes using a LiDAR sensor to measure the distance between the user and the object in the optometry device, and further steps of adjusting the tunable lens of the optometry device based on this distance), more accurate distance measurement is achieved. This results in more precise tuning of the tunable lens in the optometry device, leading to greater user comfort.

[0033] In a particularly preferred embodiment, the computer-implemented method includes the step of generating a second dataset by recording images of the scene using a camera.

[0034] The term "second dataset" refers to the camera's output. This output can be represented by images or videos. The images or videos include scenes with at least one identifiable object. In this invention, the scenes in the "second dataset" are recorded substantially in parallel with the scenes in the "first dataset." In other words, the camera and LiDAR sensor observe the same objects in the scene, but provide different datasets representing the same scene.

[0035] Furthermore, the first and second datasets include different data formats and different acquisition parameters. The first dataset, recorded by the LiDAR sensor, may include a point cloud of a scene with points at different distances identified within the scene. The first dataset can be recorded at a resolution of 940 pixels × 560 pixels and a frame rate of 30 fps. The second dataset, recorded by the camera, may include at least one image of the same scene recorded by the LiDAR sensor. The second dataset can be recorded at a resolution of 1920 pixels × 1080 pixels and a frame rate of 30 fps. Therefore, the resolution of the second dataset can be higher than the resolution of the first dataset. Alternatively, the resolution of the first dataset can be higher than the resolution of the second dataset.

[0036] If the resolution of the first dataset is not equal to the resolution of the second dataset, the dataset with the higher resolution can be downsized to the lower resolution of the other dataset. Alternatively, the dataset with the lower resolution can be interpolated to the other dataset with a higher resolution. In the example above, the resolution of the second dataset recorded by the camera can be downsized to the resolution of the first dataset recorded by the LiDAR sensor. Therefore, the resolution of the second dataset would be downsized to 940 pixels × 560 pixels.

[0037] Furthermore, the field of view of the first dataset can be at least slightly wider than that of the second dataset. In this case, the field of view of the first dataset is cropped to the field of view of the second dataset. Alternatively, the field of view of the second dataset can be at least slightly wider than that of the first dataset.

[0038] If the frame rate of the first dataset is not equal to the frame rate of the second dataset, the dataset with the larger frame rate can be downscaled to the smaller frame rate of the other dataset. In the example above, the frame rate of the second dataset recorded by the camera is equal to the frame rate of the first dataset recorded by the LiDAR sensor. Therefore, the frame rate of the second dataset will not be downscaled.

[0039] The second dataset of the computer-implemented method enables the evaluation of the recorded scene. Therefore, further steps in tuning the tunable lens of the optometry device are based on and optimized using information from the recorded scene. This results in more natural focus adjustment of the tunable lens of the optometry device.

[0040] In a particularly preferred embodiment, the computer-implemented method includes the step of generating a second dataset by recording images of the scene using an RGB camera.

[0041] The term "RGB camera" refers to a camera that provides an image matrix. Each point in the grid is assigned a color value. The resolution of the image is calculated based on the number of rows and columns. The output of the RGB camera can be represented as an image or video. The image or video includes a scene with at least one identifiable object. In this invention, the scene of the "second dataset" is recorded in parallel or substantially in parallel with the scene of the "first dataset." In other words, the RGB camera and the LiDAR sensor observe the same objects in the scene, but provide different datasets representing the same scene.

[0042] The second dataset recorded by the RGB camera enables the evaluation of the recorded scene. Therefore, further steps in tuning the tunable lens of the optometry device are based on and optimized using information from the recorded scene. This results in more natural focus adjustment of the tunable lens in the optometry device.

[0043] In a particularly preferred embodiment, the computer-implemented method includes the further step of identifying objects by evaluating a scene represented by a second dataset.

[0044] As commonly used, the term "identification" or any grammatical variation thereof refers to the identification process following the recording of an image of a scene. At the end of the identification process, objects can be identified and selected to tune the optometry device based on the distance of the identified and selected objects to the user of the optometry device.

[0045] The further step of object identification enables the computer-implemented method to not only recognize but also identify objects within the recorded scene. Therefore, this further step of object identification leads to a further step in tuning the tunable lens of the optometry device based on known and identified objects. Objects in the scene are identified by recording images with an RGB camera, and the distance from the object to the user of the optometry device is determined using data measured by a LiDAR sensor, and this distance is assigned to the object. This makes the focus adjustment of the tunable lens of the optometry device more natural and improves the user's wearing comfort.

[0046] In a particularly preferred embodiment, the computer-implemented method is characterized in that identifying the object includes the following sub-steps:

[0047] - Identify multiple objects in images of a scene represented by a second dataset, and

[0048] - Use a first database to generate an identification probability factor for each identified object among the multiple objects, and

[0049] -Based on the first database, the identification probability factor is assigned to each of the multiple objects, and

[0050] - Compare the identification probability factor of each identified object among the multiple objects with a predefined identification probability threshold, and

[0051] - Using a second database, generate a selection probability factor for each identified object among the plurality of objects that has an identification probability factor greater than the predefined identification probability threshold, and

[0052] - Compare the selection probability factor of at least each of the identified objects, which has an identification probability factor greater than the predefined identification probability threshold, with the predefined selection probability threshold.

[0053] - Determine the distance from the user of the optometry device to the identified object with the highest selection probability factor, wherein the selection probability factor of the identified object is greater than the predefined selection probability threshold.

[0054] In the first step of the identification process, multiple objects in the recorded image of the scene can be identified. As commonly used, the term "identification" or any grammatical variation thereof refers to the identification sub-step following the recording of the image of the scene. At the end of the first step of the identification process, it can be determined whether objects exist within the scene. Identification of objects within the scope of this invention can be achieved by evaluating a first dataset or preferably a second dataset.

[0055] In a further step of the identification process, an identification probability factor is generated for each of the plurality of objects using a first database, and at the end of the further step of the identification process, the identity of the identified objects in the scene is known based on a certain probability.

[0056] The term "database" in the context of this invention describes a collection of numerical methods, a collection of machine learning models, or a collection of artificial intelligence-based algorithms. In this invention, the term "first database" describes a collection of trained machine learning models for identifying objects within a scene context, said trained computer learning model including at least one identification probability factor. The object identification probability factors may be stored in tables within the first database or in any other form of information storage. Examples of a first database are TensorFlow, ImageNet, or PyTorch.

[0057] A machine learning model has been trained to recognize and identify at least one object and the context in which that object is detected. For example, the object could be a computer keyboard, and the context could be a computer keyboard on a desk. The computer keyboard and the desk, along with the surrounding environment, do indeed represent a scene. As used herein, the term "trained machine learning model" refers to a model of a machine learning model, specifically including at least one neural network trained, and in particular trained, to determine at least one identifier probability factor.

[0058] The term "identification probability factor" refers to a probability generated by a trained machine learning model by analyzing objects identified within a second dataset. The previously mentioned example includes a computer keyboard on a desk. In this example, the trained machine learning model generates an identification probability factor for the identified object being a computer keyboard on a desk. The identification probability factor can contain values ​​in the range of 0 to 1, including 0 and 1, which correspond to 0% to 100%. An identification probability factor of 0 corresponds to 0% and means that the first database does not know which object has been identified in the context. Therefore, there is no such data available to train the machine learning model to identify, for example, a computer keyboard on a desk. In this case, the machine learning model can be trained in a further step. An identification probability factor of 1 corresponds to 100% and means that the first database does indeed know with the highest possible probability which object has been identified in the context. Therefore, there is sufficient data available to train the machine learning model to identify, for example, a computer keyboard on a desk. In this case, it is not necessary to train the machine learning model in a further step. The identification probability factor preferably includes the identification probability of at least one object within the context, or alternatively, includes the identification probability factor of at least one object without context.

[0059] The term "generation" or any grammatical variation thereof refers to the process of generating certain information within the scope of this invention. This may refer, for example, to generating identifier probability factors or generating a dataset. Information can be generated through calculation, recording, extraction, assignment, derivation, processing, loading, prediction, or any other possible information generation steps.

[0060] In a further step of the identification process, identification probability factors are assigned to each of the plurality of objects based on a first database. At the end of this further step of the identification process, identification probability factors are assigned to the identified objects in the scene.

[0061] The term "assignment" or any syntactic variation thereof refers to the process of merging one piece of information with a second piece of information. For example, a second dataset can provide the existence of an object. A first database generates an identification probability factor for the object. By assigning a specific identification probability factor of a certain percentage to the identified object, a computer-implemented method knows, for example, that a computer keyboard on a table in the scene will be identified with a probability of said certain percentage. This certain percentage is in the range of 0% to 100%.

[0062] In a further step of the identification process, the identification probability factor of each identified object among the plurality of objects is compared with a predefined identification probability threshold. At the end of this further step of the identification process, for any further step, all objects with identification probability factors lower than the predefined identification probability threshold are ignored. If no identified object includes an identification probability factor greater than the predefined identification probability threshold, i) the predefined identification probability threshold can be lowered to the highest assigned identification probability factor of the identified object to ensure further processing of the computer-implemented method, or ii) the computer-implemented method stops at this step, does not tune the tunable lens, and continues to the next scene to tune the tunable lens of the optometry device. Alternatively, if no object can be identified, the tunable lens can be tuned to a default distance, for example, to infinity, or to another distance based on the raw data of the first dataset. If more than one identified object includes an identification probability factor greater than the predefined identification probability threshold, all such objects proceed to the next step of the identification process.

[0063] The term "predefined identification probability threshold" refers to a filtering step used for further processing of the identified objects. The predefined identification probability threshold can contain values ​​in the range of 0 to 1, including 0 and 1, corresponding to 0% to 100%. In a preferred embodiment, the value of the predefined identification probability threshold is between 0.5 and 0.9, corresponding to 50% to 90%. In a particularly preferred embodiment, the value of the predefined identification probability threshold is between 0.6 and 0.8, corresponding to 60% to 80%. In the most preferred embodiment, the value of the predefined identification probability threshold is 0.7, corresponding to 70%.

[0064] In a further step of the identification process, a selection probability factor is generated using a second database for each of the plurality of objects to be identified, or at least for all objects having an identification probability factor equal to or greater than an identification probability threshold. At the end of this further step of the identification process, objects to be identified are selected, the selected objects comprising a selection probability observed by the user of the optometry device.

[0065] In this invention, the term "second database" describes a collection of trained machine learning models for selecting objects within a scene context, wherein the trained computer learning models include at least one selection probability factor. The object selection probability factors may be stored in tables within the second database or in any other form of information storage. The selection probability factors in the second database are based on the user's selection probability from the optometry device.

[0066] In addition, the second database includes individual user characteristics. For example, users older than retirement age show a smaller probability factor of switching their gaze to a computer compared to younger users who work with computers daily. Younger users would show a larger probability factor of choosing a computer compared to users older than retirement age. Alternatively, a user's past behavior (taken from the second database) indicates that the user never looks at a computer. Therefore, this user's probability factor of choosing a computer is small. Other parameters affecting visual behavior (such as time of day, day of week, etc.) and thus the selected object can also be included in the second database. Examples of second databases are DeepGaze or SCEGRAM.

[0067] The machine learning model has been trained to select at least one object from the context of detected objects. For example, the objects could be a computer keyboard and a computer mouse, and the context could be a computer keyboard and a computer mouse on a desk. As used herein, the term "trained machine learning model" also refers to a model of a machine learning model, specifically including at least one neural network trained, in particular trained to determine at least one selection probability factor.

[0068] The term "selection probability factor" refers to a probability generated by a trained machine learning model by analyzing the objects(s) identified and labeled within a second dataset. The previously mentioned example includes a computer keyboard and mouse on a desk. In this example, the trained machine learning model generates selection probability factors for the identified and labeled objects. In other words, the machine learning method generates values ​​including selection probabilities for multiple objects within a scene. Selection probability factors can contain values ​​in the range of 0 to 1, including 0 and 1, which correspond to 0% to 100%. A selection probability factor of 0 corresponds to 0% and means that, based on the second database, the probability of a user focusing on an object identified and labeled in that context is nonexistent. A selection probability factor of 1 corresponds to 100% and means that, based on the second database, the probability of a user focusing on an object identified and labeled in that context is 100%. The selection probability factor preferably includes the selection probability of at least one object within the context, or alternatively, includes the selection probability of at least one object without context. Therefore, the selection probability factor for a particular object is a measure of the probability or likelihood that a user will focus on that particular object.

[0069] In a further step of the identification process, the selection probability factor of at least each identified object among the plurality of objects that has an identification probability factor greater than the predefined identification probability threshold is compared with the predefined selection probability threshold. At the end of this further step of the identification process, for each subsequent further step, all objects with selection probability factors lower than the predefined selection probability threshold are ignored.

[0070] If no identified object includes a selection probability factor greater than a predefined selection probability threshold, i) the predefined selection probability threshold can be lowered to the highest assigned selection probability factor of the identified and marked objects to ensure further processing by the computer-implemented method, or ii) the computer-implemented method stops at this step, without changing the focus of the tunable lens, and continues to the next scene to tune the tunable lens of the optometry device. If more than one identified object includes a selection probability factor greater than a predefined selection probability threshold, the object with the greater selection probability factor will proceed to the next step of the identification process.

[0071] The term "predefined selection probability threshold" refers to a filtering step used for further processing of the identified objects. The predefined selection probability threshold can contain values ​​in the range of 0 to 1, including 0 and 1, corresponding to 0% to 100%. In a preferred embodiment, the value of the predefined selection probability threshold is between 0.5 and 0.9, corresponding to 50% to 90%. In a particularly preferred embodiment, the value of the predefined selection probability threshold is between 0.6 and 0.8, corresponding to 60% to 80%. In the most preferred embodiment, the value of the predefined selection probability threshold is 0.7, corresponding to 70%.

[0072] In a further step of the identification process, based on the step of comparing the selection probability factor of at least each identified object among the plurality of objects that has an identification probability factor greater than a predefined identification probability threshold with the predefined selection probability threshold and selecting the object with the highest selection probability factor, the distance from the user of the optometry device to the identified object with the largest selection probability factor is determined. At the end of this further step of the identification process, the computer-implemented method is capable of tuning the tunable lens of the optometry device according to the distance from the user of the optometry device to the object with the highest selection probability factor based on a second dataset.

[0073] The further step of object identification enables the computer-implemented method to not only recognize but also identify objects within the recorded scene. Therefore, this further step of object identification leads to a further step in adjusting the tunable lens of the optometry device based on the known and identified objects. This makes the focus adjustment of the tunable lens of the optometry device more natural and accurate, and improves the user's wearing comfort. Furthermore, the adjustment of the tunable lens can be performed before the user shifts their gaze to another object.

[0074] In a particularly preferred embodiment, the computer-implemented method is characterized by using a machine learning algorithm to determine an identifier probability factor stored in a first database and a selection probability factor stored in a second database.

[0075] The further step of object identification enables the computer-implemented method to load identification probability factors from a first database or selection probability factors from a second database, without needing to calculate new identification or selection probability factors. This allows for faster focus adjustment of the tunable lens in the optometry device and improves user comfort. Furthermore, adjustment of the tunable lens can be performed before the user shifts their gaze to another object.

[0076] In a particularly preferred embodiment, the computer-implemented method is characterized by generating a third dataset that includes visual information of the user of the optometry device, predicting the user's next visual observation based on the third dataset, and adjusting the tunable lens of the optometry device based on the user's predicted next visual observation.

[0077] In a particular embodiment of this aspect, the term "third dataset" refers to the output of an eye-tracking sensor. This output may be represented by an RGB image or numerical information about the user's saccades and / or fixations within the context of a scene. The term "saccade" refers to the movement of at least one eye between two or more fixation phases in the same direction. The term "fixation" or "fixation point" refers to the movement behavior of the eye. During a fixation, the spatial location of the gaze remains at a single location. In other words, if a fixation is achieved, the eye does not move in a identifiable manner.

[0078] The term "visual information" refers to the user's field of vision for an optometric device and can refer to changes in the person's gaze. Based on DIN ISO 13666:2019, Section 3.2.24, the term "gaze" refers to the path from a point of interest (i.e., the point of fixation) in object space to the center of the entrance pupil of the human eye, and further includes the continuation in image space from the center of the exit pupil to the retinal fixation point (usually the fovea) of the human eye. The term "predicted next visual" refers to the user's estimated next field of vision for an optometric device.

[0079] The term "next fixation" in the context of this invention describes the end of a saccade and the beginning of a new fixation. The term "predicted next fixation" refers to the next fixation generated based on a third dataset.

[0080] The third dataset, recorded by eye-tracking sensors and implemented by the computer, enables the assessment of the user's eye movements based on the recorded scene represented by the first and second datasets. Therefore, a further step in tuning the tunable lens of the optometry device is based on and optimized using the user's predicted next fixation point. This allows for faster focus adjustment of the tunable lens and improves user comfort. Furthermore, tuning of the tunable lens can be performed before the user switches their gaze to another object.

[0081] In a particularly preferred embodiment, the computer-implemented method is characterized by predicting the user's next visual look by including the following steps:

[0082] - Determine the user's current fixation point using a third dataset, and

[0083] - Assign the distance from the object to the user of the optometry device based on the second dataset and the distance based on the object to the user of the optometry device based on the first dataset to the user's current fixation point of the optometry device, and

[0084] - Based on a third dataset, the system tracks the user's current gaze point on the optometry device until changes in eye position are detected to identify the user's next saccade.

[0085] -Predict the user's next fixation point based on the user's current gaze point and the user's identified next saccade, and

[0086] -Tune the refraction device according to the predicted next fixation point.

[0087] In the first step of the prediction process, the user's current fixation point is determined based on a third dataset. At the end of the first step of the prediction process, the fixation point is determined for the start of a new saccade (represented by time t=0).

[0088] The term “current gaze” refers to either i) a gaze that has been established at the start of the prediction process, or ii) a gaze that has been detected after the start of the prediction process.

[0089] In a further step of the prediction process, the user's current fixation point is assigned based on the object identified and selected using the second dataset and the distance from the object to the user of the optometry device, based on the first dataset. Alternatively, through further steps of the prediction process, the current fixation point is assigned based on the object identified and selected using the second dataset and the distance from the user of the optometry device to the object, based on the first dataset.

[0090] In a further step of the prediction process, the user's current gaze point is tracked based on a third dataset until the user's next saccade is identified by detecting changes in visual position. At the end of this further step of the prediction process, the start of a new saccade is detected.

[0091] The term "tracking" or any grammatical variation thereof refers to the process of identifying changes in information within the scope of this invention. This can refer to, for example, tracking a scene or visual location. Changes in information can be tracked through any other possible steps, such as observing, detecting, analyzing, searching for, or identifying changes in information.

[0092] In a further step of the prediction process, the user's next fixation point is estimated based on the user's current gaze point and the user's identified next saccade. At the end of this further step, the user's next fixation point is predicted.

[0093] The term "prediction" or any grammatical variation thereof refers to the process of estimating information within the scope of this invention. This can refer to, for example, predicting a scene or a gaze point. The estimation of information can be performed through any other step, such as calculation, interpolation, comparison, guessing, or estimation.

[0094] In a further step of the prediction process, the refraction device is tuned according to the predicted next fixation point.

[0095] The further step of identifying the object enables the computer-implemented method to estimate the user's next fixation point on the optometry device. This allows for faster focus adjustment of the optometry device's tunable lens and improves user comfort. Furthermore, the tunable lens can be adjusted before the user shifts their gaze to another object.

[0096] In a particularly preferred embodiment, the computer-implemented method is characterized in that predicting the next gaze point includes at least one of the following steps:

[0097] - Determine the gaze direction of the next identified saccade and predict the next fixation point based on the user's previous gaze information;

[0098] - Determine the gaze direction of the next identified saccade and predict the next fixation point based on the next selectable object within the determined gaze direction;

[0099] - Determine the gaze direction of the next identified saccade and predict the next fixation point based on objects within the determined gaze direction, the objects including the highest selection probability factor;

[0100] - Use the predefined next scene from the second database and assign the distance from the object to the user of the optometry device based on the first dataset, and adjust the estimate of the next fixation point based on the predefined next scene;

[0101] - Estimate the next scene by generating a fourth dataset, which includes EEG activity information of the user of the optometry device, and adjust the estimation of the next gaze based on the estimated next scene;

[0102] - The next scene is estimated by generating a fifth dataset, which includes inertial measurement information of the optometry devices (110, 710), and the estimation of the next fixation point (365, 465, 565, 665) is adjusted based on the estimated next scene.

[0103] The step of predicting the next gaze point can be achieved by determining the gaze direction of the identified next saccade and predicting the next gaze point based on the user's previous saccade information.

[0104] The term "visual direction" refers to the spatial localization detection of the current saccade. Starting from the current gaze point, the next saccade can move in any spatial direction, such as northeast. The visual direction can be specified in the form of celestial orientation, global coordinate system, or local coordinate system, or by spatial direction.

[0105] The term "previous visual information" refers to a sample of previous visual information from a user of the optometry equipment or a standard user of the optometry equipment. Previous visual information may include data on saccade distance length, saccade duration, fixation point, and fixation duration. This previous visual information is generated and stored in a third dataset.

[0106] Starting from the current fixation point and the start of the next saccade, the next fixation point is predicted by analyzing the distance length of the last saccade and / or the duration of the last saccade.

[0107] An alternative step for predicting the next gaze point can be achieved by determining the gaze direction of the identified next saccade and predicting the next gaze point based on the next selectable object within the determined gaze direction.

[0108] The term "next selectable object" refers to an object within the scanning direction. Furthermore, based on the second dataset, the next selectable object is both identifiable and selectable. Therefore, identifiable and selectable objects include an identification probability factor greater than a predefined identification probability threshold and a selection probability factor greater than a predefined selection probability threshold.

[0109] Starting from the current gaze point and the beginning of the next saccade, the next gaze point is predicted by analyzing the saccade direction. For example, if the beginning of the next saccade indicates a viewing direction in the northeast, and based on a second dataset, an object in the northeast direction is identifiable and selectable, then the next gaze point will be set at said object. If, based on the second dataset, more than one object within the saccade direction is identifiable and selectable, then the next gaze point is set on an object closer to the current gaze point. In the context of this invention, the object closer to the current gaze point is referred to as the "next selectable object."

[0110] If, based on the second dataset, no object within the saccade direction is identifiable and selectable, then i) the predefined identification probability threshold and / or the predefined selection probability threshold may be reduced to the highest assigned identification probability factor and / or the next fixation point may be set to the object with the highest assigned selection probability factor of the identified and identifiable object, to ensure further processing of the computer-implemented method, or ii) the computer-implemented method may stop at this step without changing the focus of the tunable lens and continue to the next scene to tune the tunable lens of the optometry device.

[0111] An alternative step in predicting the next fixation point can be achieved by determining the gaze direction of the identified next saccade and predicting the next fixation point based on objects within the determined gaze direction, including the objects with the highest selection probability factor.

[0112] Starting from the current gaze point and the beginning of the next saccade, the next gaze point is predicted by analyzing the saccade direction. For example, if the beginning of the next saccade indicates a gaze direction in the northeast, and based on a second dataset, multiple objects in the northeast direction are identifiable and selectable, then the next gaze point will be set at the object with the highest selection probability factor among the multiple objects included in the gaze direction.

[0113] If, based on the second dataset, no object within the scanning direction is identifiable and selectable, then i) the predefined identification probability threshold and / or the predefined selection probability threshold can be reduced to the highest assigned identification probability factor and / or the highest assigned selection probability factor of the identified and identified objects to ensure further processing by the computer-implemented method, or ii) the computer-implemented method stops at this step, without changing the focus of the tunable lens, and continues to the next scene to tune the tunable lens of the optometry device.

[0114] An alternative step for predicting the next fixation point can be achieved by using a predefined next scene from a second database, assigning the distance from the object to the optometry device based on the first dataset, and adjusting the estimation (or prediction) of the next fixation point based on the predefined next scene.

[0115] The term "predefined next scene" describes an item in the second database. This item is configured to train a machine learning model or to provide a test scene for the user of the optometry device. Such a test scene is displayed on a screen (e.g., a monitor or mobile device display). The scene is evaluated by a method implemented by the computer before the "predefined next scene" is displayed to the user of the optometry device. Before the user of the optometry device identifies the predefined next scene, identification probability factors and selection probability factors for objects within the scene are determined. Therefore, the tunable lens of the optometry device is tuned based on the scene before the user identifies the predefined next scene.

[0116] An alternative step in predicting the next fixation point can be achieved by generating a fourth dataset to estimate the next scene, which includes the user's electroencephalogram (EEG) information from the optometry device; and adjusting the estimation (or prediction) of the next fixation point based on the estimated next scene.

[0117] Specifically, in view of this aspect, the term "fourth dataset" refers to the sensor output including information on brain electrical activity (such as data from electroencephalography or electrooculography). This output can be represented by voltage signals from different regions of brain activity, including information about the amplitude and frequency of said voltage signals, and provides information about the user's next gaze and next fixation point.

[0118] The starting point for estimating the next scene includes detecting the user's current gaze using the optometry device, for example, the right side of the user's visual field. If the user decides to shift their gaze to the left side of their visual field, brain activity signals are sent to the user's eyes. These brain activity signals are detected and analyzed by sensors used to generate a fourth dataset.

[0119] Furthermore, the tunable lens of the optometry device is tuned based on the detected and analyzed brain activity signals from the fourth dataset. Therefore, the user's gaze follows the tuning step of the tunable lens.

[0120] An alternative step for predicting the next fixation point can be achieved by generating a fifth dataset to estimate the next scene, which includes inertial measurement information from the optometry device; and adjusting the estimation (or prediction) of the next fixation point based on the estimated next scene.

[0121] In particular, in view of this aspect, the term "fifth dataset" refers to the output of the inertial measurement unit, which includes inertial measurement information (such as the angular rate and orientation of the optometry device obtained by using a combination of accelerometers, gyroscopes, and perhaps magnetometers) and provides information about the user's next gaze and next fixation point.

[0122] The starting point for estimating the next scene includes the user's current gaze detected by the optometry device and the inertial measurement information associated with the user's current gaze. If the user decides to switch their gaze from one side to the other, a change in the fifth dataset is generated. This change in the fifth dataset is detected and analyzed by the inertial measurement unit used to generate it.

[0123] Furthermore, the tunable lens of the optometry device is tuned based on the detected and analyzed inertial measurement information from the fifth dataset. Therefore, the user's gaze point follows the tuning step of the tunable lens.

[0124] The first aspect of the present invention is completely solved by a method implemented by a computer of the type described above.

[0125] In a second aspect, the present invention relates to a system comprising:

[0126] - Optometry equipment

[0127] - A first device, configured to generate a first dataset, the first dataset including the distance from the user of the optometry device to the object.

[0128] - A control unit configured to tune the optometry device based on the user-to-object distance of the optometry device according to a first dataset.

[0129] The system is characterized by its primary device being a LiDAR sensor.

[0130] As commonly used, the term "system" refers to at least one or more devices (particularly connected via a computer network). These devices can be connected via a network using at least one connection interface on any one of the devices.

[0131] The term "control unit" refers to a device configured to adjust optometric equipment based on a dataset generated by other devices. An example of a "control unit" is an Optotune device driver controlled by a PC or Raspberry Pi.

[0132] In a particularly preferred embodiment, the system includes a second device, wherein the second device is a camera.

[0133] In a particularly preferred embodiment, the system includes a second device, wherein the second device is an RGB camera.

[0134] In a particularly preferred embodiment of the system, the control unit is configured to tune the optometry device based on an object identified in a second dataset recorded by the second device.

[0135] In a particularly preferred embodiment, the system includes a third device, which is an eye tracker device configured to predict the user's next gaze based on a third dataset.

[0136] In a particularly preferred embodiment of the system, the control unit is configured to select an object based on an identification probability factor and a selection probability factor determined as described above with reference to the method implemented by the computer.

[0137] The second aspect of the invention is completely solved by a system of the type described above. Attached Figure Description

[0138] Further features, characteristics, and advantages of the invention will become clear from the following description of exemplary embodiments of the invention taken in conjunction with the accompanying drawings.

[0139] Figure 1 A first exemplary embodiment, showing a top view of a user of an optometric device with an adjustable lens, is illustrated.

[0140] Figure 2 A block diagram of a first exemplary embodiment of the method for operating a tunable lens using a LiDAR sensor and an RGB camera is shown.

[0141] Figure 3 A block diagram of a second exemplary embodiment of the method for operating a tunable lens using a LiDAR sensor, an RGB camera, and an eye-tracking device is shown.

[0142] Figure 4 illustrates various scenarios to illustrate a first exemplary embodiment of the prediction process.

[0143] Figure 5 illustrates various scenarios to demonstrate a second exemplary embodiment of the prediction process.

[0144] Figure 6 illustrates various scenarios to illustrate a third exemplary embodiment of the prediction process.

[0145] Figure 7 A cross-sectional view of a second exemplary embodiment of an optometric device with an adjustable lens is shown.

[0146] exist Figure 1 The image shows a first exemplary embodiment of a system 100 for operating a tunable lens 101. The system 100 is worn by a user 110 and includes a tunable lens 101, a LiDAR sensor 103, an RGB camera 105, an eye-tracking device 117, and a control unit located within the tunable lens 101. Figure 1 (Not shown in the image).

[0147] like Figure 1 As shown, LiDAR sensor 103 and RGB camera 105 simultaneously record scene 107 observed by user 110. Scene 107 may include several objects 109, 111, and 115. The dataset recorded by LiDAR sensor 103 includes data representing the corresponding distances from user 110 to each of objects 109, 111, and 115. The dataset recorded by RGB camera 105 includes data representing images of each object 109, 111, and 115. Eye-tracking device 117 records the movement of user 110's eyes based on the observed scene 107.

[0148] Based on a second dataset recorded by RGB camera 105, the control unit of system 100 identifies, marks, and selects objects 107 and 109. Furthermore, based on a first dataset recorded by LiDAR sensor 103, the control unit of system 100 determines the corresponding distance from user 110 to each of objects 107 and 109. In the next step, the control unit of system 100 assigns each distance to the corresponding object 109, 111, and 115. Based on the control unit's selection and the determined distances to the selected objects, the tunable lens 101 is tuned according to the data from the LiDAR sensor and the RGB camera.

[0149] Optionally, the data from the eye-tracking device 117 can be processed by the control unit of the system 100 and used to tune the tunable lens 101 based on the data from the LiDAR sensor, the RGB camera, and the eye-tracking device.

[0150] Figure 2 The process steps of method 200 for operating tunable lenses 101, 701 by using LiDAR sensors 103, 703 and RGB cameras 105, 705 of systems 100 and 700 are shown in a first exemplary embodiment.

[0151] In the first data generation step 201 of the first exemplary embodiment of method 200, a first dataset 211 and a second dataset 212 are generated. The first dataset 211 is generated by a LiDAR sensor, and the second dataset 212 is generated by an RGB camera. Both the first dataset 211 and the second dataset 212 include the same scene 107 (see Figure 1 The scene includes objects 109, 111, and 115.

[0152] Based on the second dataset 212, the identification step 203 is performed. In the first sub-step of the identification step 203, an identification probability factor 283 is determined for each of objects 109, 111 and 115 in scenario 107 based on the second dataset 212 and the first database 281.

[0153] In an exemplary embodiment, object 109 may include an identification probability factor 283 of 0.8, object 111 may include an identification probability factor 283 of 0.7, and object 115 may include an identification probability factor 283 of 0.6.

[0154] In the second sub-step of identification step 203, the identification probability factor 283 of each identified object 109, 111, and 115 is compared with a predefined identification probability threshold 285. At the end of the second sub-step of identification step 203, for further steps, objects with an identification probability factor 283 lower than the predefined identification probability threshold 285 are ignored. In the above exemplary embodiment, the predefined identification probability threshold 285 may include a value of 0.7. Therefore, for further steps, objects 109 and 111 with an identification probability factor 283 equal to or greater than 0.7 will be considered, and for further steps, object 115 with an identification probability factor 283 of 0.6 will not be considered.

[0155] Based on the second dataset 212 and the identification step 203, the selection step 205 is performed.

[0156] In the first sub-step of selection step 205, based on the second dataset 212 and the second database 291, selection probability factors 293 are assigned to the remaining objects 109 and 111 of scenario 107 that have identification probability factors equal to or greater than the identification probability threshold. In the above exemplary embodiment, object 109 may include a selection probability factor 293 of 0.8, and object 111 may include a selection probability factor 293 of 0.7.

[0157] In the second sub-step of selection step 205, the selection probability factor 293 of each identified object 109 and 111 is compared with a predefined selection probability threshold 295. At the end of this second sub-step of selection step 205, for further steps, objects with a selection probability factor 293 lower than the predefined selection probability threshold 295 are ignored. If more than one object exceeds the predefined selection probability threshold 295, then objects 297 with a higher or highest selection probability factor 293 are selected. In the exemplary embodiment described above, the predefined selection probability threshold 295 may include a value of 0.7. Therefore, for further steps, objects 109 and 111 with a selection probability factor 293 equal to or higher than 0.7 may be considered. The selection probability factor 293 of object 109 is higher than that of object 111. Therefore, for further steps, object 109 will be considered.

[0158] Based on the first dataset 211 and the selected object 297, the assignment step 207 is performed. In the assignment step 207, the distance of the selected object 297 is assigned to the selected object 297 based on the first dataset 211.

[0159] Based on and following the assignment step 207, a tuning step 209 is performed. In tuning step 209, the tunable lens 101 is tuned based on the selected object 297 and the distance between the selected object 297 and the first dataset 211. This results in a tuned tunable lens 101.

[0160] Figure 3 A second exemplary embodiment of method 300 for operating a tunable lens by using a LiDAR sensor 103, an RGB camera 105, and an eye-tracking device 117 is shown.

[0161] In the data generation step 301 of the second exemplary embodiment of method 300, a first dataset 311, a second dataset 312, and a third dataset 313 are generated. The first dataset 311 is generated by the LiDAR sensor 103, the second dataset 312 is generated by the RGB camera 105, and the third dataset 313 is generated by the eye-tracking device 117. Both the first dataset 311 and the second dataset 312 include the same scene 107, which includes objects 109, 111, and 115 (see...). Figure 1 The third dataset 313 includes user 110's saccade and gaze information within the context of scene 107. In other words, the first dataset 311 and the second dataset 312 represent scene 107, while the third dataset 313 represents the movement of the user's eyes triggered by scene 107.

[0162] Based on the third dataset 313, a determination step 315 is performed. In the determination step 315, the current gaze point 361 of the user 110 of the tunable lens 101 is obtained based on the third dataset 313. In an exemplary embodiment, the current gaze point 361 is set to the object 109 on the left side of the identified scene 207.

[0163] Based on the current gaze point 361 and the third dataset 313, a change in the gaze position 362 is detected through tracking step 317. A change in gaze position 363 can lead to a new gaze point. In the exemplary embodiment described above, the gaze position changes from the left side of the identified scene 107 to the right side of the identified scene 107.

[0164] Based on the change in gaze position 363 detected by tracking step 317, the estimated gaze point 365 is estimated by prediction step 319. In this exemplary embodiment, the estimated gaze point 365 is set to the object 107 to the right of the identified scene 107.

[0165] Based on the second dataset 312, an identification step 303 similar to the identification step 203 of the previous first exemplary embodiment for operating the tunable lens and a selection step 305 similar to the selection step 205 of the previous first exemplary embodiment for operating the tunable lens are performed. At the end of the selection step 305, one or more optional objects 397, including a selection probability factor larger than a predefined selection probability threshold, can be applied to set the next fixation point.

[0166] Based on the first dataset 311, the available objects 397 based on the second dataset 312, and the estimated gaze point 365, an assignment step 307 is performed. In this assignment step 307, the estimated distance from the gaze object 107 to the user 110 is assigned to the new gaze point 365, based on the first dataset 311 and the estimated position of the gaze object 107. In other words, if the object 107 has a selection probability factor greater than a predefined selection probability threshold, the computer-implemented method generates information in assignment step 307 to assign the estimated gaze point 365 to the object 107.

[0167] Based on assignment step 307, tuning step 309 is performed. In tuning step 309, the tunable lens 101 is tuned based on the estimated gaze object 107 and the distance of the estimated gaze object 107 based on the first dataset 311. This results in the tuned tunable lens 101.

[0168] With the aid of Figure 4, a first exemplary embodiment of the prediction process 400 based on this method is explained in more detail. Figure 4 is subdivided into Figure 4a )to Figure 4cEach subdivision map represents the same scene observed by the user at different times t1 to t3. This first exemplary embodiment is based on determining the gaze direction of the identified next saccade and predicting the next fixation point based on the next optional object within the determined gaze direction.

[0169] Figure 4a Scene 407, including objects 409, 411, and 415, is shown, and the prediction process begins at time t1 = 0 s. It is assumed that the current gaze point 461 is set at object 409 at said time t1 = 0 s. The current gaze point 461 is represented as a circle encompassing all possible microsagitations within the gaze point. The current gaze point 461 will remain set at object 409 as long as it at least partially covers object 409.

[0170] Figure 4b Scene 407, including objects 409, 411, and 415, is shown, and the current saccade 463 during the prediction process at time t2 = 0.5 s is shown. The current saccade 463 is along direction d1 (see...) at said time t2 = 0.5 s. Figure 4b The arrow in the image moves. The current scan 463 will continue moving until the user of the tunable lens focuses on a new object.

[0171] Based on the detected saccades, the computer-implemented method predicts the user's new fixation point 465 by determining the direction d1 of the current saccade 463 and detecting the next object within the direction d1 of the current saccade 463, the next object including i) an identification probability factor not lower than a predefined identification probability threshold and ii) a selection probability factor not lower than a predefined selection probability threshold.

[0172] In a digital example of an exemplary first embodiment of the prediction process 400, it is assumed that a predefined identification probability threshold is set to 0.7, and a predefined selection probability threshold is set to 0.7. If an object 415 on the detected saccade direction d1 includes an identification probability factor of 0.8 and a selection probability factor of 0.8, and another object 411 on the detected saccade direction d1 includes an identification probability factor of 0.8 and a selection probability factor of 0.9, then both objects 415 and 411 are suitable for setting the predicted fixation point 465.

[0173] Figure 4cScene 407, including objects 409, 411, and 415, is shown, and the prediction process ends at time t3 = 1.0 s. At time t3 = 1.0 s, the predicted gaze point 465 is set at object 415. Since object 415 is the next selectable object within the saccade direction d1 at time t2, the predicted gaze point 465 is set at object 415. If object 415 is not in the saccade direction d1 at time t2, the predicted gaze point 465 will be set at object 411.

[0174] A second exemplary embodiment of the prediction process 500 based on this method is explained in more detail with the help of Figure 5. Figure 5 is subdivided into Figure 5a )to Figure 5c Each subdivision map represents the same scene observed by the user at different times t1 to t3. This second exemplary embodiment is based on determining the gaze direction of the identified next saccade and predicting the next fixation point based on objects within the determined gaze direction, the objects including the highest selection probability factor.

[0175] Figure 5a Scene 507, including objects 509, 511, and 515, is shown, and the prediction process begins at time t1 = 0 s. It is assumed that the current gaze point 561 is set at object 509 at said time t1 = 0 s. The current gaze point 561 is represented as a circle encompassing all possible microsaccades within the gaze point. The current gaze point 561 will remain set at object 509 as long as it at least partially covers object 509.

[0176] Figure 5b Scene 507, including objects 509, 511, and 515, is shown, and the current scan 563 during the prediction process at time t2 = 0.5 s is represented. At time t2 = 0.5 s, the current scan 563 moves along direction d1 (see...). Figure 5b (The arrow in the image). The current scan 563 will continue to move until the user of the adjustable lens focuses on a new object.

[0177] Based on the detected saccades, the computer-implemented method predicts a new fixation point 565 of the user by determining the direction d1 of the current saccade 563 and detecting objects within the direction d1 of the current saccade 563, including i) an identification probability factor not lower than a predefined identification probability threshold and ii) a selection probability factor not lower than a predefined selection probability threshold, and in the case of multiple objects, including the highest selection probability of said object.

[0178] Similar to the example described with reference to Figure 4, assume that the predefined label probability threshold is set to 0.7, and assume that the predefined selection probability threshold is set to 0.7. If object 515 includes a label probability factor of 0.8 and a selection probability factor of 0.8, and object 511 includes a label probability factor of 0.8 and a selection probability factor of 0.9, then both object 515 and object 511 are suitable for setting the predicted fixation point 565.

[0179] Figure 5c Scene 507, including objects 509, 511, and 515, is shown, and the prediction process ends at time t3 = 1.0 s. At time t3 = 1.0 s, the predicted gaze point 565 is set at object 511. Since object 511 is a selectable object within the saccade direction d1 at time t2, and this selectable object includes the highest selection probability factor, the predicted gaze point 565 is set at object 511. If object 511 is not in the saccade direction d1 at time t2, the predicted gaze point 565 will be set at object 515.

[0180] A third exemplary embodiment of the prediction process 600 based on this method is explained in more detail with the help of Figure 6. Figure 6 is subdivided into Figure 6a )to Figure 6c Each subdivision map represents the same scene observed by the user at different times t1 to t3. This third exemplary embodiment is based on determining the gaze direction of the identified next saccade and predicting the next fixation point based on the user's previous gaze information.

[0181] Figure 6a Scene 607, including objects 609, 611, and 615, is shown, and the prediction process begins at time t1 = 0 s. It is assumed that the current gaze point 661 is set at object 609 at said time t1 = 0 s. The current gaze point 661 is represented as a circle encompassing all possible microsaccades within the gaze point. The current gaze point 661 will remain set at object 609 as long as it at least partially covers object 609.

[0182] Figure 6b Scene 607, including objects 609, 611, and 615, is shown, and the current saccade 663 during the prediction process is represented at time t2 = 0.5 s. At time t2 = 0.5 s, the current saccade 663 moves along direction d1. The current saccade 663 will continue to move until the user of the tunable lens looks at a new object. Based on the user's previous visual information, the estimated length of the current saccade 663 is estimated via vector d2.

[0183] Based on the detected saccades, the computer-implemented method predicts the user's fixation point 665 by determining the direction d1 of the current saccade 663 and estimating the length of the current saccade 663 using the length of vector d2. Adding vector d2 to fixation point 661 at time t1 = 0 s yields the predicted fixation point 665. The predicted fixation point 665 is set if i) the predicted fixation point 665 at least partially covers an object within direction d1, and ii) the object includes an identification probability factor not less than a predefined identification probability threshold, and iii) the object includes a selection probability factor not less than a predefined selection probability threshold.

[0184] Similar to the example described with reference to Figures 4 and 5 in the third embodiment of prediction process 600, it is assumed that the predefined label probability threshold is set to 0.7, and it is also assumed that the predefined selection probability threshold is set to 0.7. If it is assumed that object 615 includes a label probability factor of 0.8 and a selection probability factor of 0.8, and that object 611 includes a label probability factor of 0.8 and a selection probability factor of 0.9, then both object 615 and object 611 are suitable for setting the predicted fixation point 665.

[0185] Figure 6c Scene 607, including objects 609, 611, and 615, is shown, and the prediction process ends at time t3 = 1.0 s. At time t3 = 1.0 s, a predicted gaze point 665 is set at object 615 based on vector d2. Since object 615 is covered by the predicted gaze point 665 based on vector d2 and is located within the saccade direction d1 at time t2, the predicted gaze point 665 is set at object 615. Because the distance of object 665 from the actual gaze point 609 does not conform to the length and direction of vector d2, object 611 will not be selected for setting the predicted gaze point 665. Therefore, object 611 is not covered by the predicted gaze point 665, even though object 611 is located within the saccade direction d1 at time t2 and includes a higher selection probability factor (compared to object 615).

[0186] Figure 7 An optometry device 700 with an adjustable lens 701 is shown, which can be operated according to the method of the invention. The optometry device 700 is configured to be worn by a user 710 and includes a first adjustable lens 701a for the user's right eye R and a second adjustable lens 701b for the user's left eye L. Both adjustable lenses 701a and 701b are arranged in a frame 731 configured to be worn by the user 710. In an alternative embodiment, the frame 731 may be part of the housing of the optometry device 700 configured to determine the optometric prescription values ​​for the user 710.

[0187] The eyeglass frame 731 includes a LiDAR sensor 703, an RGB camera 705, and one or two eye-tracking devices 717a and 717b, one for detecting movement of the user 710's right eye (R) and the other for detecting movement of the user 710's left eye (L). The viewing directions of the LiDAR sensor 703 and camera 705 are directed towards the viewing directions of the user 710's right eye (R) and left eye (L). The optics of the LiDAR sensor 703 and RGB camera 705 are configured such that both devices record data of nearly identical scenes. The detection directions of the eye-tracking devices 717a and 717b are almost opposite to the viewing directions of the LiDAR sensor 703 and camera 705, i.e., towards the user 710's eyes (R and L).

[0188] The optometry device 700 also includes a control unit 730 that receives data from a LiDAR sensor 703, an RGB camera 705, and eye-tracking devices 717a and 717b. The control unit 730 generates control data for the tunable lenses 701a and 701b, as described in more detail above. Additionally, the control unit 730 generates output data describing the required correction values ​​for the user 710's eye's R and L viewing abilities, which can be output to and displayed on a monitor or any other suitable output device 732. The control unit 730 may also be mounted on the frame 731, but it may also be mounted separately from the frame 731, for example, in a console for operating the optometry device.

Claims

1. A computer-implemented method (200, 300) for operating an optometry device (700), the method comprising the steps of: - generating (201, 301) a first data set (211, 311) comprising information about a distance of a user (110, 710) of the optometry device (700) to at least one object (109, 111, 115) comprised in a scene (107, 407, 507, 607), wherein the first data set (211, 311) is generated by using a LiDAR sensor (103, 703) that measures the distance of the user (110, 710) of the optometry device (700) to the at least one object (109, 111, 115) by evaluating the scene (107, 407, 507, 607) represented by the first data set (211, 311) characterized in that - generating (201, 301) a second data set (212, 312) by recording an image of the scene (107, 407, 507, 607) by a camera, wherein the first data set (211, 311) and the second data set (212, 312) both comprise the same scene (107, 407, 507, 607), - identifying (203, 303) the at least one object (109, 111, 115) by a control unit (730) by evaluating the scene (107, 407, 507, 607) represented by the second data set (212, 312), - generating (201, 301) a third data set (303) comprising visual information of the user (110, 710) of the optometry device (700) by an eye tracking sensor, - predicting (400, 500, 600) a next visual of the user (110, 710) based on the third data set (303), and - tuning (209, 309) an adjustable lens (101, 701) of the optometry device (700) based on the predicted next visual of the user (110, 710) and the distance to the identified object comprised in the first data set (211, 311).

2. The computer-implemented method (200, 300) of claim 1, wherein, The recording of the image is performed by using an RGB camera (105, 705).

3. The computer-implemented method (200, 300) of claim 1, wherein, The identifying (203, 303) of the at least one object comprises: - recognizing a plurality of objects (109, 111, 115) in the image of the scene (107, 407, 507, 607) represented by the second data set (212, 312), and - generating an identification probability factor (283) for each recognized object (109, 111, 115) of the plurality of objects (109, 111, 115) using a first database (281), and - selecting the identified object (109, 111, 115) having the highest identification probability factor (283) as the at least one object (109, 111, 115). - assigning the identification probability factor (283) to each object (109, 111, 115) of the plurality of objects (109, 111, 115) based on the first database (281), and - comparing the identification probability factor (283) of each identified object (109, 111, 115) of the plurality of objects (109, 111, 115) with a predefined identification probability threshold (285), and - generating a selection probability factor (293) for each identified object (287, 387) of the plurality of objects (109, 111, 115) having an identification probability factor (283) equal to or greater than the predefined identification probability threshold (285) using a second database (291), and - comparing the selection probability factor (293) of at least each identified object (287, 387) of the plurality of objects having an identification probability factor (283) equal to or greater than the predefined identification probability threshold (285) with a predefined selection probability threshold (295), and - determining a distance of the user (110, 710) of the optometry device (700) to the identified object (287, 387) having the greatest selection probability factor (293), wherein the selection probability factor (293) of the identified object (287, 387) is greater than the predefined selection probability threshold (295).

4. The computer-implemented method (200, 300) of claim 3, wherein, The identification probability factors (283) stored in the first database (281) and the selection probability factors (293) stored in the second database (291) are determined using a machine learning algorithm.

5. The computer-implemented method (200, 300) according to one of the claims 1 to 4, characterized in that, Predicting (400, 500, 600) the next visual of the user (110, 710) comprises the following steps: - determining (315) a current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700) based on the third data set (303), and - assigning objects based on the second data set (212, 312) and distances of the objects to the user (110, 710) of the optometry device (700) based on the first data set (211, 311) to the current gaze point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700), and - tracking (317) the current gaze point (361, 461, 561, 616) of the user (110, 710) of the optometry device (700) based on the third data set (303) until a next saccade (363, 463, 563, 663) of the user (110, 710) of the optometry device (700) is identified by detecting a change in the visual position, and - determining (319) a next visual of the user (110, 710) of the optometry device (700) based on the first data set (211, 311) and the second data set (212, 312) based on the tracked current gaze point (361, 461, 561, 616) of the user (110, 710) of the optometry device (700) and the identified next saccade (363, 463, 563, 663) of the user (110, 710) of the optometry device (700). - predicting (319) a next fixation point (365, 465, 565, 665) of the user (110, 710) of the optometry device (700) based on the current fixation point (361, 461, 561, 661) of the user (110, 710) of the optometry device (700) and the identified next saccade (363, 463, 563, 663) of the user (110, 710) of the optometry device (700), and - tuning (209, 309) the optometry device (700) according to the predicted next fixation point (365, 465, 565, 665).

6. The computer-implemented method (200, 300) of claim 5, wherein, The prediction (400, 500, 600) of the next fixation point (365, 465, 565, 665) comprises at least one of the following steps: - determining a visual direction (dl) of the identified next saccade (363, 463, 563, 663) and predicting the next fixation point (365, 465, 565, 665) based on previous visual information (d2) of the user (110, 710), - determining a visual direction (dl) of the identified next saccade (363, 463, 563, 663) and predicting the next fixation point (365, 465, 565, 665) based on a next selectable object within said determined visual direction (dl), - determining a visual direction (dl) of the identified next saccade (363, 463, 563, 663) and predicting the next fixation point (365, 465, 565, 665) based on an object (109, 111, 115) within said determined visual direction (dl), said object (109, 111, 115) comprising the highest selection probability factor (293), - using a predefined next scene of the second database (291) and assigning a distance of the object to the user (110, 710) of the optometry device (700) based on the first data set (211, 311) and adjusting the estimation of the next fixation point (365, 465, 565, 665) based on the predefined next scene, - estimating a next scene by generating a fourth data set comprising electroencephalographic activity information of the user (110, 710) of the optometry device (700) and adjusting the estimation of the next fixation point (365, 465, 565, 665) based on the estimated next scene, and - estimating a next scene by generating a fifth data set comprising inertial measurement information of the optometry device (110, 710) and adjusting the estimation of the next fixation point (365, 465, 565, 665) based on the estimated next scene.

7. A system (100) comprising: - an optometry device (700), - a first device configured to generate a first dataset (211, 311) comprising a distance of a user (110, 710) of the optometry device (700) to at least one object (109, 111, 115) comprised in a scene (107, 407, 507, 607), wherein the first device is a LiDAR sensor (103, 703), - a control unit (730) configured to tune (209, 309) a tunable lens (101, 701) of the optometry device (700), characterized in that - a second device configured to generate a second dataset (212, 312) by recording an image of the scene (107, 407, 507, 607), wherein the second device is a camera, wherein the first dataset (211, 311) and the second dataset (212, 312) both comprise the same scene (107, 407, 507, 607), - a third device configured to generate a third dataset (303) comprising visual information of the user (110, 710) and to predict a next visual of the user (110, 710) based on the third dataset (303), wherein the third device is an eye tracker device (117, 717a, 717b), and wherein - the control unit (730) is configured to tune (209, 309) the tunable lens (101, 701) of the optometry device (700) based on the predicted next visual of the user (110, 710) and the distance to the identified object comprised in the first dataset (211, 311).

8. The system of claim 7, wherein, The camera is an RGB camera (105, 705).

9. The system according to one of claims 7 or 8, characterized in that The control unit (730) is configured to select the object (109, 111, 115) based on an identification probability factor (283) and a selection probability factor (293).

Images