Systems and methods for providing visual simulation for intraocular lens patients

Abstract

Systems and methods are presented for providing a visual simulation of a patient with an eye disorder. A synthetic image representing a real-world scene is obtained. The synthetic image includes a plurality of image layers, where each image layer represents an object in the real-world scene at a particular viewing distance. A first eye model representing the patient's eye optical system is generated. A second eye model representing the viewer's eye optical system is generated. The second eye model is modified by performing a mathematical function. A simulated image representing the patient's vision is generated by convolving the first eye model and the modified second eye model with the synthetic image. In some embodiments, a tone mapping algorithm may also be applied to the simulated image to simulate a night scene.

Description

BACKGROUND OF THE DISCLOSURE TECHNICAL FIELD

[0002] This disclosure relates to visual simulation, and more particularly to simulating the vision of a patient implanted with an intraocular lens according to various embodiments of the disclosure.

[0003] Description of the prior art

[0004] Cataract is a condition in which a cloudy area forms on the lens of a patient's eye, thereby obstructing the patient's vision. Cataracts are common in the elderly. Typically, cataracts can be cured by replacing the natural (cataractous) lens of the eye with an intraocular lens (IOL). Although a patient may have a pre-existing eye condition (e.g., presbyopia, etc.), a patient who has an IOL implanted in the eye after cataract surgery can regain the ability to see fine spatial details at various distances (e.g., far, near, and intermediate distances, etc.) by using an IOL (e.g., a multifocal IOL) with appropriate refractive power and characteristics for that patient.

[0005] To illustrate the performance of various IOLs implanted in a patient, it is useful to present a simulated vision of a patient who is about to undergo cataract surgery so that a user (e.g., the patient, others providing care to the patient) can decide which IOL to use when undergoing cataract surgery. However, generating a visual simulation of a patient can be challenging because the simulated image (which simulates the patient's perception of a real-world scene) is presented on a flat screen (e.g., a tablet display, etc.) and then is intended to pass through the user's eyes before being aligned by the user. Therefore, there is a need in the art to provide a visual simulator that accurately simulates the vision of an intraocular lens patient. SUMMARY OF THE DISCLOSURE

[0006] According to some embodiments, a system includes a non-transitory memory and one or more hardware processors configured to read instructions from the non-transitory memory to cause the system to perform operations, the operations including: accessing a first eye model representing a patient's eye and a second eye model representing an eye of a user different from the patient; obtaining a composite image including a plurality of image layers corresponding to a scene, wherein each image layer of the plurality of image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying the second eye model by performing a mathematical function on the second eye model; and generating a simulated image by applying the first eye model and the modified second eye model to each image layer of the plurality of image layers of the composite image.

[0007] According to some embodiments, a method includes: generating, by one or more hardware processors, a first eye model based on biometric information of a patient's eye; accessing, by the one or more hardware processors, a second eye model representing the eye of a user different from the patient; obtaining, by the one or more hardware processors, a synthetic image including a plurality of image layers corresponding to a scene, wherein each of the plurality of image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying, by the one or more hardware processors, the first eye model based on the second eye model; and generating, by the one or more hardware processors, a simulated image by applying the first eye model to each of the plurality of image layers of the synthetic image.

[0008] According to some embodiments, a non-transitory machine-readable medium stores machine-readable instructions that, when executed, cause a machine to perform operations including: accessing a first eye model representing a patient's eye and a second eye model representing the eye of a user different from the patient; obtaining a synthetic image including a plurality of image layers corresponding to a scene, wherein each of the plurality of image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying the second eye model by performing a mathematical function on the second eye model; and generating a simulated image by applying the first eye model and the modified second eye model to each of the plurality of image layers of the synthetic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a more complete understanding of the present technology, its features and advantages, reference is made to the following description in conjunction with the accompanying drawings.

[0010] Figure 1 is a diagram of a system for generating a simulated image according to some embodiments.

[0011] Figure 2A illustrates a process for generating a simulated image representing a daytime scene according to some embodiments.

[0012] Figure 2B illustrates a process for generating a simulated image representing a nighttime scene according to some embodiments.

[0013] Figure 3 illustrates a synthetic image according to some embodiments.

[0014] Figure 4A illustrates the human perception of a real-world scene according to some embodiments.

[0015] Figure 4B illustrates the double-pass effect according to some embodiments.

[0016] Figure 5A Shows another synthetic image representing a daytime scene according to some embodiments.

[0017] Figure 5B Shows another synthetic image representing a nighttime scene according to some embodiments.

[0018] Figure 6A and Figure 6B is a diagram of a processing system according to some embodiments.

[0019] In the drawings, elements having the same reference numerals have the same or similar functions. Detailed Description

[0020] The description and drawings showing aspects, embodiments, implementations, or modules of the present invention should not be considered limiting - it is the claims that define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the invention. Similar numerals in two or more figures represent the same or similar elements.

[0021] In this description, specific details are set forth that describe some embodiments consistent with this disclosure. To provide a thorough understanding of the embodiments, numerous specific details are set forth. However, it will be clear to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not limiting. Those skilled in the art can implement other elements that are not specifically described herein but are within the scope and spirit of this disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment can be incorporated into other embodiments, unless otherwise specifically stated or if the one or more features would render the embodiment inoperative.

[0022] The techniques described below relate to systems and methods for providing visual simulations for patients having an eye disorder (such as a cataract disorder, a presbyopia disorder) or patients implanted with an intraocular lens (IOL) (also referred to as IOL patients). In some embodiments, the visual simulation system is configured to present images that simulate how a patient would perceive real-world scenes after a cataract surgery in which their opaque natural lens is replaced with an IOL. For example, the visual simulation system can present the simulated images on a digital device (e.g., a flat panel display) for viewing by one or more users.

[0023] To generate a simulated image that mimics the vision of a patient, a vision simulation system can first prepare a composite image with multiple layers of objects at different viewing distances. The composite image can be generated by combining multiple image layers of real-world scenes, where different image layers represent objects at different viewing distances. Thus, by combining the image layers, the composite image can represent a three-dimensional real-world scene. In some embodiments, the composite image is calibrated according to how the patient's eyes would view the real-world scene. That is, all objects in the composite image are set to a viewing angle defined by the assumed physical size of the object and the viewing distance.

[0024] Then, the vision simulation system can establish (e.g., obtain or generate) an eye model (e.g., a first eye model) representing the patient's eyes. In some embodiments, the first eye model can include one or more point spread functions (PSFs). In such embodiments, the PSF represents how the patient's eyes focus light from objects at different viewing distances based on the patient's eye optical properties when forming a retinal image. Then, the vision simulation system can modify (e.g., blur) the corresponding layer by convolving the first eye model with each layer in the composite image, and blend all the blurred layers together to form a simulated image. The simulated image represents the realization of the real-world scene on the retina of the patient's eyes through the eye optical system. In some embodiments, the vision simulation system can use alpha blending to combine the blurred layers into a single plane to handle object occlusion within the real-world scene. Applying the first eye model to the images of the real-world scene can generate the vision perceived by the eye patient. However, as discussed above, since the simulated image is presented on a flat display in front of the user's eyes, the user's eye optical system needs to be considered when generating the simulated image to avoid the double-pass effect.

[0025] When a viewer sees the simulated image shown on the display, a double-pass effect may occur in the viewer's eyes (e.g., the user's eyes). The double-pass effect mentioned herein describes the phenomenon that the simulated image generated by applying a first eye model associated with the eye patient to the real-world scene is further processed by the eye optical system of the viewer's eyes before the viewer aligns with it. Thus, when the viewer aligns with the image, the simulated image is perceived as being further processed (e.g., blurred) by the eye optical system of the viewer (e.g., the user). To enable the viewer to perceive what the patient actually perceives when viewing the real-world scene (e.g., to eliminate the double-pass effect), the influence of the viewer's eye optical system needs to be subtracted from the simulated image before presenting the simulated image on the display.

[0026] Thus, in some embodiments, the visual simulation system can establish another eye model (e.g., a second eye model) representing the eye optical system of the viewer's eye (e.g., the user's eye). The visual simulation system can then modify the second eye model by performing a mathematical function on the second eye model (e.g., performing an inverse function on the second eye model). The visual simulation system can then cancel (e.g., eliminate) the double-pass effect by further applying the modified second eye model to the simulated image. For example, when modifying a synthetic image, the visual simulation system can modify (e.g., blur) the corresponding layer by convolving the first eye model and the modified second eye model with each layer in the synthetic image, and then blend the blurred layers together to form a simulated image.

[0027] The visual simulation system can then transmit and / or present the simulated image on a display device (e.g., a flat panel display). Note that since the synthetic image is calibrated based on the viewing distance of the object such that each pixel in the synthetic image faces a specific vision, the user is required to view the simulated image at a preset magnification and at a distance corresponding to the calibration of the synthetic image (e.g., 40 cm). Thus, when presenting the simulated image to the user, the visual simulation system can also indicate to the user to view the image at a specific viewing distance associated with the simulated image.

[0028] The above techniques are effective in simulating a patient's vision of a daytime scene. For a nighttime scene, the visual simulation system needs to consider additional factors, such as the potentially higher dynamic range of the nighttime scene, the human perception of light intensity (e.g., brightness) in the dark, and the influence of light source intensity (e.g., may cause visual impairments such as halos, starbursts, etc.). Thus, to simulate a patient's vision of a nighttime scene, the synthetic image obtained for the nighttime scene needs to have a high dynamic range (HDR) such that the pixel intensity is linearly related to the physical intensity of the light source. The visual simulation engine can first process the HDR synthetic image using the techniques described herein related to simulating a daytime scene (e.g., by convolving the first eye model and the modified second eye model with the HDR synthetic image). The visual simulation system can further process the simulated image by applying a tone mapping algorithm to the synthetic image to compress the intensity range in the HDR image to a standard (or low) dynamic range such that the visibility of the spatial features in the HDR image is retained and the image can be appropriately presented on a display device having only standard (or low) dynamic range display capabilities. In some embodiments, the visual simulation system can first segment the image into a portion corresponding to the light source and a portion corresponding to non-light sources, and apply the tone mapping algorithm only to the light source portion of the simulated image.

[0029] Figure 1FIG. 0 shows a system 100 according to some embodiments, within which a visual simulation system as discussed herein may be implemented. System 100 includes a synthetic image generator 130 for generating synthetic images (e.g., a conventional synthetic image for a daytime scene or an HDR synthetic image for a nighttime scene, etc.), a visual simulation engine 102 for modifying the synthetic image based on one or more eye models to generate a simulated image, and a display device 140 for presenting the simulated image to a user. As shown, the visual simulation engine 102 includes a simulation manager 104, one or more patient eye models 106, one or more user eye models 108, an image blending module 110, and a nighttime simulation module 112. In some embodiments, the simulation manager 104 may obtain information about a patient's eye biometric information (e.g., pupil diameter, anterior chamber depth, corneal refractive power, etc.) by retrieving it from a diagnostic device (not shown) or from a database 114. The simulation manager may then select a specific intraocular lens (IOL) for the simulated image from a plurality of IOLs. For example, lens manufacturers may produce different IOLs with different optical designs and / or different refractive powers. In some embodiments, the simulation manager 104 may select a specific IOL from a plurality of IOLs produced by a lens manufacturer based on the patient's biometric information. For example, the simulation manager 104 may use one or more machine learning models to predict the best lens type for a patient to achieve a target visual quality based on the patient's biometric information. In some embodiments, the simulation manager 104 may also present a list of a plurality of IOLs on the display device 140 such that a user (or physician) can select a specific IOL for generating the simulated image. The simulation manager may then generate a first eye model (e.g., the patient's eye model) based on the biometric information and the selected IOL.

[0030] The simulation manager 104 may also generate a second eye model (e.g., the user eye model). In some embodiments, the simulation manager 104 may obtain (e.g., via a diagnostic device, etc.) the biometric information of the user's eye that will view the simulated image. Instead of measuring the user's biometric information, the simulation manager 104 in some embodiments may simply generate a generic eye model based on the eyes of ordinary people that can be viewed by any user to avoid having to measure the biometric information for different users. The simulation manager 104 may then use the image modification module 110 and / or the nighttime simulation module 112 to modify the synthetic image based on the first eye model and the second eye model to generate the simulated image, and then may present the simulated image on the display device 140.

[0031] Figure 2AProcess 200 for providing a simulated image of a daytime scene to a user in accordance with an embodiment of the present disclosure is shown. The simulated image represents a daytime scene as perceived by a patient (e.g., through the patient's eye optical system). In some embodiments, process 200 may be executed by the visual simulation engine 102. As discussed above, the patient may have different eye conditions, such as a cataract condition, a presbyopia condition, an intraocular lens eye condition (e.g., a patient who has undergone cataract surgery to replace the natural lens of the eye with an IOL). In some embodiments, process 200 generates a simulated image of how the patient would perceive a real-world scene after a cataract surgery in which their opaque natural lens is replaced with an IOL.

[0032] Process 200 begins by obtaining (at step 205) a composite image that includes a plurality of image layers. For example, the simulation manager 104 may obtain the composite image from the composite image generator 130. The composite image may be generated by combining a plurality of image layers of a real-world scene, where each image layer represents an object at a different viewing distance. Figure 3 A composite image 304 generated by combining different image layers 306a - 306c is shown. In some embodiments, the different image layers 306a - 306c represent different objects at different viewing distances from the real-world scene. For example, in the different image layers 306a - 306c, image layer 306a represents an object at a far viewing distance (e.g., a mountain, a background), image layer 306b represents an object at an intermediate viewing distance (e.g., a map), and image layer 306c represents an object at a near viewing distance (e.g., a compass). Thus, by combining the image layers 306a - 306c, the composite image 304 represents a three-dimensional real-world scene. In some embodiments, the composite image is calibrated according to how the eye of the eye patient would view the real-world scene. That is, all objects in the composite image are set to a viewing angle defined by the assumed physical size of the object and the viewing distance. For calibration, each object (the objects in each layer) takes on a real-world physical size (e.g., height 物体物理 ) and a predefined viewing distance (l 观看距离 ). The viewing angle at which the object faces can be calculated using the following equation:

[0033]

[0034] All layers of the composite image (e.g., composite image 306) are scaled to have the same radians per pixel using the following equation:

[0035]

[0036] where height 物体像素 is the height of the object, in pixels.

[0037] Figure 5A Shows another composite image 502 representing a daytime driving scene by combining different image layers 504-510. In this example, image layer 504 represents objects at a far viewing distance (e.g., other vehicles, roads, other background objects), image layer 506 represents objects at an intermediate viewing distance (e.g., 75 cm) (e.g., the vehicle's dashboard), image layer 508 represents objects at another intermediate viewing distance (e.g., 60 cm) (e.g., the vehicle's computer screen), and image layer 510 represents objects at a near viewing distance (e.g., 40 cm) (e.g., a coffee cup).

[0038] Figure 4A Shows how a patient may perceive a real-world scene 402. Light from the real-world scene first passes through the patient's eye optical system 404, where different parts of the real-world scene 402 (e.g., light reflected from different parts of the real-world scene 402) may be blurred due to the eye optical system 404 to form a retinal image 406. The retinal image 406 then reaches the patient's brain, which can perform additional processing (e.g., increasing contrast, neural adaptation, etc.) on the retinal image 406 to generate a neural perception 408 of the real-world scene 402. Thus, to simulate how the retinal projection of an object in the real-world scene is blurred by the optical system of the patient's eye, a first eye model representing the optical system of the patient's eye is obtained. In this way, at step 210, process 200 establishes a first eye model representing the patient's eye.

[0039] The optical characteristics of the human eye can be described by the wavefront of the entire eye. The wavefront is characterized at the pupil (e.g., cornea) plane of the eye, which measures the total optical aberration of the human eye. To simulate the wavefront of an artificial lens eye implanted with an IOL, a double-surface reduced eye model can be constructed. The double-surface reduced eye model includes two surfaces: one surface for the cornea of the patient's eye and one surface for the IOL. The corneal surface may have higher-order aberrations, and the surface for the IOL includes an optical diffraction structure by the design of the IOL. In some embodiments, the biometric information (e.g., pupil diameter, anterior chamber depth, corneal refractive power, etc.) represented by the eye model can be obtained by acquiring measurement results of the patient's eye (e.g., using a diagnostic device). In some embodiments, the biometric information can be obtained based on clinical data collected from artificial lens eyes that have previously undergone cataract surgery. For example, the average values of these biometric characteristics from the clinical data can be used to represent an average artificial lens eye. The corneal astigmatism and the toric component of the IOL surface can be combined at the corneal plane in the eye model. Defocus (blur) can be varied at the corneal plane based on different viewing distances in order to calculate the vision of an IOL patient at different viewing distances. Table 1 below shows exemplary parameters of the double-surface reduced eye model.

[0040]

[0041] Table 1: Key parameters of the double-surface reduced eye model.

[0042] In some embodiments, the optical performance of the human eye can be quantified using various optical metrics calculated directly from the wavefront of the eye or from the point spread function (PSF). The PSF represents the transformation associated with the wavefront at the image plane of the eye (e.g., the retina of the eye), which defines how the patient's eye focuses (and / or blurs) light from objects at different viewing distances based on the optical characteristics of the patient's eye when forming a retinal image. The PSF can be calculated using units of angular perspective in object space (radians). The Fourier transform of the PSF can yield the optical transfer function (OTF) or the modulation transfer function (MTF) of the eye. While the PSF is in the spatial domain, the OTF expresses the PSF in the spatial frequency domain. On the other hand, the MTF is the modulus of the OTF, which is related to the sensitivity of the human eye to contrast stimuli at different spatial frequencies. The PSF, OTF, and MFT all describe the characteristics of the eye optical system when forming a retinal image (e.g., retinal image 406).

[0043] In some embodiments, the first eye model can be generated using the PSF based on the biometric information of the patient's eye (or biometric information calculated using clinical data). For example, the simulation manager 104 can generate the first eye model (e.g., the patient eye model 106) by calculating the PSF using the biometric information. The simulation manager 104 can then add defocus (e.g., blur) and corneal astigmatism at the corneal plane in the first eye model. The amount of defocus added to the first eye model can be proportional (or inversely proportional) to the viewing distance. Additionally, the amount of corneal astigmatism added at the corneal plane in the first eye model may depend on whether the patient's intraocular lens eye is implanted with an aspheric or toric IOL. In some embodiments, the anterior corneal higher order aberration (HOA) is not used to generate the first eye model. The diffractive optical design profile of the IOL can be expressed as a front wave and projected onto the corneal plane from the IOL plane using scaling. The scaling factor γ 患者眼睛 can be calculated using the following equation:

[0044]

[0045] The incident pupil size can be set according to the lighting conditions of the expected real-world scene. The Stiles-Crawford effect of the eye can be included in the calculation as pupil tapering. In some embodiments, monochromatic light with a wavelength of 550 nm can be used to calculate the PSF.

[0046] Once the first eye model (e.g., the patient eye model 160) is generated, the simulation manager 104 can store the first eye module 160 in the database 114. In some embodiments, to simulate how the retinal projection of an object in a real-world scene (e.g., the real-world scene 402) becomes blurred through the optical system of the patient's eye, the simulation manager 104 can use the first eye model 160 (which represents the eye optical system of the patient's eye) - e.g., the PSF calculated using the techniques described herein - to modify the synthetic image (e.g., the synthetic image 304) to form a simulated image. For example, the simulation manager 104 can convolve the first eye model (e.g., the PSF) with the corresponding image layers (e.g., the image layers 306a - 306c) in the synthetic image, as shown in the following equation:

[0047]

[0048] where the symbol represents image convolution, and the superscript 'layer' represents the calculation performed for each layer in the synthetic image.

[0049] Then, the simulation manager 104 can generate a simulated image by successively combining different modified image layers (e.g., each image layer is modified by convolving the first eye model 160 with the corresponding image layer) according to a hypothetical viewing distance from far to near. In some embodiments, to ensure that the simulated image reflects the object occlusion and boundaries of the real-world scene 402, the simulation manager 104 can use an alpha blending method to combine different modified image layers. For example, the simulation manager 104 can calculate the alpha value for blending by convolving the alpha channel for each image layer with the corresponding PSF, as shown in the following equation:

[0050]

[0051] In some embodiments, the simulation manager 104 can perform alpha blending on different modified image layers according to a hypothetical viewing distance from far to near using the following equation:

[0052] Alpha 前一层 = 1 - Alpha 当前层 ………………………………………………(6)

[0053] However, as discussed herein, the simulated image generated by convolving the first eye model 160 with the image layer using the above techniques may not capture additional visual processing within the human visual system, such as potential neural adaptation for obtaining enhanced contrast perception and higher neural transfer function (NTF) gain. Additionally, when a viewer views the simulated image, the simulated image also does not account for the effects of the viewer's eye optical system. As Figure 4B shown, the simulated image 412 is generated based on the first eye model 160, which simulates how a patient perceives the real-world scene 402 (e.g., the perceived image 406). However, when a user views the simulated image 412, the simulated image 412 will pass through the user's eye optical system 414 and be further processed by the user's brain before forming the user's perception 416 of the simulated image 412. Since the simulated image is viewed through the viewer's eye optical system, there will be additional image contrast loss, especially at high spatial frequencies, which results in a reduced evaluation of the spatial sharpness of the vision of the patient after using the IOL. This degradation of image quality is generally referred to as the double-pass effect. Therefore, the user's perception of the simulated image 416 is different from the simulated image 412.

[0054] When a viewer sees a simulated image shown on a display, a double-pass effect may occur in the viewer's eyes (e.g., the user's eyes). The double-pass effect mentioned in this article describes the phenomenon that when a simulated image generated by applying a first eye model associated with an eye patient to a real-world scene is viewed by a viewer, it is further processed (e.g., blurred) by the eye optical system of the viewer's eyes before the viewer aligns. Therefore, how the viewer perceives the simulated image may be different from the simulated image. Thus, when the viewer aligns the image, the simulated image is perceived as being further processed (e.g., blurred) by the eye optical system of the viewer (e.g., the user), as shown in the following equation:

[0055]

[0056] As shown in Equation (7), due to the double-pass effect, the image formed at the back of the viewer's eyes is different from the image simulated for the patient's eyes. Therefore, in order for the viewer of the simulated image to perceive what the patient actually perceives when viewing a real-world scene (e.g., eliminate the double-pass effect), it is necessary to subtract the influence of the viewer's eye optical system from the simulated image before presenting the simulated image on the display (e.g., pre-compensate for the optical blur of the viewer's eyes in the simulated image).

[0057] First, the simulated image can be generated by the simulation manager 104 using the following equation, which does not consider the eye optical system of the viewer:

[0058]

[0059] where, represents the Fourier transform, and represents the inverse Fourier transform. OTF 患者眼睛 is the optical transfer function of the patient's eyes, i.e., the Fourier transform of PSF 患者眼睛 .

[0060] To correct for the double-pass effect, the simulation manager 104 of some embodiments may generate a second eye type (e.g., user eye model 108) representing the viewer's eye optical system. Thus, process 200 obtains (at step 215) a second eye model representing the user's eye. In some embodiments, the simulation manager 104 may obtain (e.g., via a diagnostic device, etc.) biometric information of the user's (e.g., viewer's) eye that will view the simulated image. Instead of measuring the biometric information of each individual user, the simulation manager 104 of some embodiments may simply generate a generic eye model based on the eyes of "average" people (e.g., based on previously obtained clinical data related to patients with healthy eyes (normal vision within a predetermined eye refractive power range)). Then, the simulation manager 104 may generate a second eye model 108 based on the viewer's biometric information using the techniques described herein that refer to generating a patient eye model 106. For example, the second eye model 108 may also be expressed as one or more PSFs, which may also be converted to an OTF and / or MTF.

[0061] To account for the viewer's eye optical system, the simulated image may be generated by the simulation manager 104 using the following equation:

[0062]

[0063] In some embodiments, due to the variation in eye aberrations between different viewers, the simulation manager 104 may utilize the modulation transfer function (MTF) of the average human eye MTF 普通观看者眼睛 to replace the OTF 患者眼睛 , as shown in the following equation:

[0064]

[0065] Note that if the biometric information of the viewer is obtained, the MTF 观看者眼睛 may be utilized to replace the MTF 普通观看者眼睛 . Although using the MTF of an average viewer ignores the individual variations in both the phase component and the amplitude component of the OTF 观看者眼睛 , since the MTF 普通观看者眼睛 describes how the contrast at different spatial frequencies is transmitted through the optical system of an average viewer's eye, it retains the key optical features relevant to the simulation. The reciprocal of the second eye model in equation (10) (e.g., 1 / MTF 普通观看者眼睛 ) compensates for the average contrast loss caused by the double-pass effect of the viewer's eye. Equation (10) may also be expressed as follows:

[0066]

[0067] where PSF双通校正 It can be expressed as:

[0068]

[0069] Return reference Figure 2A , Process 200 then generates (at step 220) a simulated image by modifying the composite image based on the first eye model and the second eye model. For example, the simulation manager 104 may use the image modification module 110 to modify the composite image (e.g., composite image 304) based on the first eye model 106 and the second eye model 108 to generate a simulated image. In some embodiments, the image modification module 110 may use equations (11) and (12) to generate the simulated image, e.g., by convolving the first eye model 106 (e.g., PSF 患者眼睛 ) and the reciprocal of the second eye model (e.g., 1 / MTF 普通观看者眼睛 ) with each image layer in the composite image 304 and blending the modified image layers.

[0070] In addition to the eye's optical system of the viewer, neural adaptation also plays a role in how the viewer perceives the simulated image. Neural adaptation is the ability of the human visual system to adapt to visual blur. Specifically, when the retinal image (e.g., retinal image 406) is transmitted to the human brain, the brain can adapt to the visual blur caused by the eye's optical system (or other factors) by automatically increasing the contrast sensitivity. In some embodiments, the simulation manager 104 may model the neural adaptation of the average human eye as an increase in the neural transfer function gain. Then, the simulation manager 104 uses the same technique mentioned above for two-pass correction to apply the effect of neural adaptation to the simulated image. For example, the simulation manager 104 may directly apply the neural adaptation model to the composite image (in addition to the first eye model and the second eye model) (e.g., convolving the neural adaptation model with the composite image) to generate the simulated image. In some embodiments, instead of generating another model, the simulation manager 104 may simply apply the contrast increase effect of neural adaptation in the second eye model before applying the PSF of the two-pass effect to the composite image.

[0071] Process 200 then presents (at step 225) the simulated image to the user via the display device and instructs (at step 230) the user to view the presented image at a specific distance associated with the first eye model. For example, the simulation manager 104 can transmit the simulated image to the display device 140 for display on the display device 140. Note that the two-pass correction is accurate only when the display device is placed at a specific viewing distance to recreate the same viewing conditions for the patient, such that the viewer can experience the spatial scale in the real-world scene (e.g., real-world scene 402). As discussed above, the synthetic image (e.g., synthetic image 304) is calibrated such that each pixel faces a specific viewing angle of the display device of a specific screen size and is viewed at a specific viewing distance. Thus, the simulated image needs to be shown at a specific magnification (e.g., the viewing distance from the viewer's eyes) such that the physical viewing angle of the image pixels on the display screen facing the viewer's eyes is the same as the physical viewing angle of the patient's eyes assumed in the simulation. In other words, the viewing angle of the objects in the image facing the viewer is the same as the viewing angle of the objects in the real-world scene facing the patient. For example, the simulation manager 104 can present the simulated image at a predetermined magnification and can instruct (e.g., via the display device 140) the viewer to view the simulated image at a viewing distance of 40 cm.

[0072] The above techniques are effective in simulating the patient's vision of daytime scenes. For night scenes, the visual simulation system needs to consider additional factors, such as the potentially higher dynamic range of night scenes, the human perception of light intensity (e.g., luminance) in the dark, and the effects of light source intensity (e.g., may cause visual impairments such as halos, starbursts, etc.). Thus, in order to simulate the patient's vision of night scenes, the synthetic image obtained for night scenes needs to have a high dynamic range (HDR) such that the pixel intensity is linearly related to the physical intensity of the light source. Figure 2B Process 232 for generating a simulated image representing the patient's perception of a night scene according to an embodiment of the present disclosure is shown. In some embodiments, process 232 can be executed by the visual simulation engine 102. Process 232 begins with obtaining (at step 235) a synthetic image representing a night scene.

[0073] Figure 5BDisplays a synthetic image 520 of a night driving scene. Similar to the synthetic image 304 representing a daytime scene, the synthetic image 520 also includes multiple image layers, with each image layer representing an object at a different viewing distance. For example, the synthetic image 520 may include an image layer representing an object at a far viewing distance (e.g., a road, an oncoming vehicle, etc.), an image layer representing an object at an intermediate viewing distance (e.g., the dashboard and screen in a vehicle), and an image layer representing an object at a near viewing distance (e.g., a coffee cup). The synthetic image 520 is a high dynamic range (HDR) image because the pixel values in the synthetic image 520 represent the true physical light intensity in the night scene. For example, the pixels in the synthetic image 520 representing light sources (e.g., vehicle headlights, streetlights, etc.) have pixel values representing a light intensity (e.g., illuminance) of 10000 cd / m 2 or higher magnitude, while the pixels representing non-light sources have pixel values representing a light intensity of 1 cd / m 2 magnitude.

[0074] Process 232 then generates (at step 240) a simulated image based on the synthetic image. For example, the simulation manager 104 can use the techniques described herein related to simulating a daytime scene (as described herein with reference to Figure 2A described) to process the HDR synthetic image (e.g., synthetic image 520). For example, the simulation manager 104 can establish a first eye model representing the patient's eyes ( Figure 2A step 210), obtain a second eye model representing the viewer's eyes ( Figure 2A step 215), and generate a simulated image by modifying the synthetic image 520 based on the first eye model and the second eye model (e.g., using equations (12) and (13)). Since the (multiple) PSFs in the first eye model of the eye optical system representing the patient's eyes take into account how light (e.g., generated by the light sources in the scene) spreads when passing through the patient's eye optical system, appropriate halos can be produced on the simulated image by convolving the synthetic image with the first eye model.

[0075] In some embodiments, the simulation manager 104 may use the night simulation module 112 to further process the simulated image. For example, the simulation manager 104 may apply a tone mapping algorithm to the composite image 520 to compress the intensity range in the HDR image to a standard (or low) dynamic range so that the simulated image can be appropriately presented on a display device having only a standard (or low) dynamic range display capability. In some embodiments, the simulation manager 104 may first segment the image into a portion corresponding to a light source (e.g., portion 522) and a portion corresponding to a non-light source, and apply the tone mapping algorithm only to the light source portion 522 of the simulated image. Process 232 thus identifies (at step 240) a first region of the light source in the simulated image that represents a night scene, modifies (at step 245) the simulated image by applying the tone mapping algorithm to at least the first region of the simulated image, and combines (at step 250) the first region and the (multiple) remaining regions of the simulated image. In one particular example, the tone mapping algorithm defines a cutoff illuminance value (e.g., 10000 cd / m 2 ), such that any pixel having an illuminance value higher than the cutoff illuminance value (e.g., pixels within the first region) is mapped to the maximum luminance value (e.g., pixel value 255) of the final simulated image, and pixels having an illuminance value lower than the cutoff illuminance value are mapped to different luminance values according to a curve (e.g., linearly, non-linearly, etc.). After applying the tone mapping algorithm to the simulated image, process 232 presents (at step 260) the simulated image on the display device. For example, the simulation manager 104 may transmit the simulated image to the display device 140 for display on the display device 140.

[0076] In addition to simulating the vision of a patient with an IOL replacing their natural lens, the vision simulation engine 102 of some embodiments may also present simulated images simulating the vision of patients having other types of eye conditions such as cataract eye conditions, presbyopia eye conditions. To simulate the vision of a patient with a cataract, a first eye model may be generated by the simulation manager 104 to represent the eye optical system of an eye having a cataract condition. For example, the simulation manager 104 may model the wavefront of a cataract eye as a rough surface having a spatially varying phase delay. The wavefront roughness (both the amplitude of the spatial variation and the spatial frequency of that variation) used in the simulation increases with the severity of the cataract. The yellowing of the cataract lens and the reduction in the light transmissibility through the lens are simulated by gradually reducing the RGB pixel values of the simulated image.

[0077] Figure 6A And Figure 6B is a diagram of a processing system according to some embodiments. Although Figure 6A and Figure 6BTwo embodiments are shown, but those of ordinary skill in the art will also readily appreciate that other system embodiments are possible. According to some embodiments, Figure 6A and / or Figure 6B The processing system of Figure 6A and / or Figure 6B may represent a computing system that may be included in one or more of the following: the lens selection platform 102 and the ECP devices 130, 140, and 150, etc.

[0078] Figure 6A Computing system 600 is shown, wherein the components of system 600 are electrically connected to each other using bus 605. System 600 includes a processor 610 and a system bus 605 that couples each of the system components (including memories in the form of, for example, read-only memory (ROM) 620, random access memory (RAM) 625, etc. (e.g., PROM, EPROM, FLASH-EPROM, and / or any other memory chip or cartridge)) to processor 610. System 600 may further include a cache 612 of high-speed memory that is directly connected to, adjacent to, or integrated as part of processor 610. System 600 may access data stored in ROM 620, RAM 625, and / or one or more storage devices 630 through cache 612 for high-speed access by processor 610. In some examples, cache 612 may provide a performance boost to avoid latency when processor 610 accesses data previously stored in cache 612 from memory 615, ROM 620, RAM 625, and / or the one or more storage devices 630. In some examples, the one or more storage devices 630 store one or more software modules (e.g., software modules 632, 634, 636, etc.). Software modules 462, 634, and / or 636 may control and / or be configured to control processor 610 to perform various actions, such as the processes of method 300 and / or 500. And although system 600 is shown as having only one processor 610, it should be understood that processor 610 may represent one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), etc. In some examples, system 400 may be implemented as a stand-alone subsystem, and / or implemented as a board added to a computing device, or implemented as a virtual machine.

[0079] To enable a user to interact with system 600, system 600 includes one or more communication interfaces 640 and / or one or more input / output (I / O) devices 645. In some examples, the one or more communication interfaces 640 may include one or more network interfaces, network interface cards, etc., to provide communication according to one or more network and / or communication bus standards. In some examples, the one or more communication interfaces 440 may include interfaces for communicating with system 600 via a network (such as network 115). In some examples, the one or more I / O devices 645 may include one or more user interface devices (e.g., keyboard, pointing / selection device (e.g., mouse, touchpad, roller, trackball, touchscreen, etc.), audio device (e.g., microphone and / or speaker), sensor, actuator, display device, etc.).

[0080] Each of the one or more storage devices 630 may include non-transitory non-volatile storage provided, for example, by a hard disk, optical medium, solid state drive, etc. In some examples, each of the one or more storage devices 630 may be co-located with system 600 (e.g., a local storage device) and / or remote from system 600 (e.g., a cloud storage device).

[0081] Figure 6B A computing system 650 based on a chipset architecture is shown, which can be used to perform any of the methods described herein (e.g., method 300 and / or 510). System 650 may include a processor 655, which represents any number of physically and / or logically distinct resources capable of executing software, firmware, and / or other computations, such as one or more CPUs, multi-core processors, microprocessors, microcontrollers, DSPs, FPGAs, ASICs, GPUs, TPUs, etc. As shown, processor 655 is assisted by one or more chipsets 660, which may also include one or more CPUs, multi-core processors, microprocessors, microcontrollers, DSPs, FPGAs, ASICs, GPUs, TPUs, coprocessors, encoder-decoders (CODECs), etc. As shown, the one or more chipsets 660 interface processor 655 with one or more of the I / O devices 665, one or more storage devices 670, memory 675, bridge 680, and / or one or more communication interfaces 690. In some examples, the one or more I / O devices 665, one or more storage devices 670, memory, and / or one or more communication interfaces 690 may correspond to Figure 6A and the similarly named counterparts in system 600.

[0082] In some examples, the bridge 680 may provide additional interfaces for providing the system 650 access to one or more user interface (UI) components, such as one or more keyboards, pointing / selection devices (e.g., mouse, touchpad, trackball, trackball, touch screen, etc.), audio devices (e.g., microphone and / or speaker), display devices, and the like. According to some embodiments, the system 600 and / or 650 may provide a graphical user interface (GUI) that is adapted to assist a user (e.g., a surgeon and / or other medical staff) in performing the processes of method 200 and / or 232.

[0083] The methods according to the above embodiments may be implemented as executable instructions stored on a non-transitory tangible machine-readable medium. When executed by one or more processors (e.g., processor 610 and / or processor 655), the executable instructions may cause the one or more processors to perform one or more of the processes of method 200 and / or 210. Some common forms of machine-readable media that may include the processes of method 200 and / or 210 are, for example, floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic medium, CD-ROM, any other optical medium, punched cards, paper tapes, any other physical medium with a hole pattern, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.

[0084] The apparatus for implementing the methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include laptop computers, smart phones, small personal computers, personal digital assistants, and the like. A portion of the functions described herein may also be embodied in peripheral devices and / or add-on cards. By further example, such functions may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0085] Although the illustrative embodiments have been shown and described, various modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without correspondingly using other features. Those of ordinary skill in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the following claims, and it is appropriate that the claims be broadly construed in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A system, comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations, the operations including: accessing a first eye model representing a patient's eye and a second eye model representing the eye of a user different from the patient, wherein the first eye model represents an intraocular lens eye having a specific multifocal intraocular lens, and wherein the first eye model is associated with a plurality of magnifications and a plurality of viewing distances based on a viewing angle; obtaining a synthetic image including a plurality of image layers corresponding to a scene, wherein each of the plurality of image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying the second eye model by performing a mathematical function on the second eye model; generating a simulated image by applying the first eye model and the modified second eye model to each of the plurality of image layers of the synthetic image; and presenting the simulated image on a display device at a specific magnification of the plurality of magnifications and instructing the user to view the simulated image presented on the display device at a specific viewing distance of the plurality of viewing distances.

2. The system of claim 1, wherein the scene is a daytime scene, and wherein modifying the second eye model includes generating a reciprocal of the second eye model.

3. The system of claim 1, wherein the operations further include calibrating the synthetic image by: for each object in each of the plurality of image layers, determining a viewing angle based on the associated real-world dimension and real-world viewing distance.

4. The system of claim 1, wherein the operations further include generating the first eye model at least in part based on characteristics of the specific multifocal intraocular lens.

5. A method, comprising: generating, by one or more hardware processors, a first eye model based on biometric information of a patient's eye, wherein the first eye model represents an intraocular lens eye having a specific multifocal intraocular lens, and wherein the first eye model is associated with a plurality of magnifications and a plurality of viewing distances based on a viewing angle; accessing, by the one or more hardware processors, a second eye model representing the eye of a user different from the patient; obtaining, by the one or more hardware processors, a synthetic image including a plurality of image layers corresponding to a scene, wherein each of the plurality of image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying, by the one or more hardware processors, the first eye model based on the second eye model; generating, by the one or more hardware processors, a simulated image by applying the first eye model to each of the plurality of image layers of the synthetic image; and Present the simulated image on a display device at a specific magnification among the multiple magnifications and instruct the user to view the simulated image presented on the display device at a specific viewing distance among the multiple viewing distances.

6. The method according to claim 5, wherein, generating the simulated image further includes blending the multiple image layers.

7. The method according to claim 5, wherein, the scene is a night scene, and wherein the method further includes: identifying a first region in the simulated image representing a light source of the scene; and modifying the simulated image by applying a tone mapping algorithm to pixels within the first region of the simulated image.

8. The method according to claim 7, wherein, the composite image is a high dynamic range image, wherein each pixel in the composite image represents a real-world illuminance intensity, and wherein the method further includes converting the high dynamic range image to a standard dynamic range image associated with the display device.

9. The method according to claim 5, wherein, the first eye model includes a point spread function.

10. The method according to claim 5, wherein, generating the simulated image includes convolving the first eye model with each of the multiple image layers.

11. A non-transitory machine-readable medium having machine-readable instructions stored thereon, the machine-readable instructions being executable to cause a machine to perform operations, the operations including: accessing a first eye model representing a patient's eye and a second eye model representing an eye of a user different from the patient, wherein the first eye model represents an intraocular lens eye having a specific multifocal intraocular lens, and wherein the first eye model is associated with multiple magnifications and multiple viewing distances based on a viewing angle; obtaining a composite image including multiple image layers corresponding to a scene, wherein each of the multiple image layers includes an object associated with a real-world dimension and a real-world viewing distance in the scene; modifying the second eye model by performing a mathematical function on the second eye model; generating a simulated image by applying the first eye model and the modified second eye model to each of the multiple image layers of the composite image; and presenting the simulated image on a display device at a specific magnification among the multiple magnifications and instructing the user to view the simulated image presented on the display device at a specific viewing distance among the multiple viewing distances.

12. The non-transitory machine-readable medium according to claim 11, wherein, the scene is a day scene, and wherein modifying the second eye model includes generating the reciprocal of the second eye model.

13. The non-transitory machine-readable medium according to claim 11, wherein, the scene is a night scene, and wherein the operations further include: identifying a first region in the simulated image representing a light source of the scene; and modifying the simulated image by applying a tone mapping algorithm to pixels within the first region of the simulated image.

Images