Facilitating intraocular lens alignment using automatic detection of purkinje images

Abstract

A system includes an ophthalmic microscope comprising a camera and a controller coupled to the camera. The controller is configured to receive at least one image from the camera, the at least one image comprising a representation of a patient's eye. The controller segments the at least one image using a machine learning model to obtain at least one segmented image, the at least one segmented image comprising one or more labels of one or more Purkinje images represented in the at least one image. The controller generates an output based on the at least one segmented image. The output may be an estimated location of the visual axis of the eye. The output may include an estimate of the offset between the visual axis and the center of an intraocular lens (IOL) implanted in the eye. The segmented image may further label the IOL and one or more possible anatomical features of the eye.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 570,407, filed March 27, 2024, which is incorporated herein by reference in its entirety. Background Technology

[0002] This disclosure generally relates to providing imaging during ophthalmic surgeries, such as cataract surgery.

[0003] The human eye receives light through a transparent outer part called the cornea and focuses the resulting image onto the retina through the lens. The quality of the focused image depends on many factors, including the size and shape of the eye and the transparency of the cornea and lens. When the lens becomes less transparent due to age or disease, vision declines because less light is transmitted to the retina. This defect in the eye's lens is medically known as a cataract. Additionally, the lens may lose its ability to adjust with age, a condition called presbyopia. An acceptable treatment for these conditions is surgical removal of the lens, which is then replaced with an artificial lens (IOL).

[0004] Facilitating the implementation of cataract surgery and other treatments would be an advancement in this field. Summary of the Invention

[0005] In some embodiments, a system includes an ophthalmic microscope comprising at least one camera and a controller coupled to the at least one camera. The controller is configured to receive at least one image from the at least one camera, the at least one image comprising a representation of a patient's eye. The controller uses a machine learning model to segment the at least one image to obtain at least one segmented image, the at least one segmented image comprising one or more labels of one or more Purkinje images represented in the at least one image. The controller generates output based on the at least one segmented image. Attached Figure Description

[0006] To gain a detailed understanding of the features described above, reference can be made to the embodiments for a more specific description of the disclosure, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only exemplary embodiments and should not be construed as limiting the scope of the disclosure, and may allow for other equally effective embodiments.

[0007] Figure 1 An example ophthalmic system for performing ophthalmic treatments in a surgical setting, according to certain embodiments, is shown.

[0008] Figure 2A Components of an ophthalmic microscope according to certain embodiments are shown, along with Purkinje images.

[0009] Figure 2B The arrangement of the lights in an ophthalmic microscope is shown.

[0010] Figure 2C The reflection of Purkinje images from an intraocular lens (IOL) is shown according to certain embodiments.

[0011] Figure 3 Example IOL is shown.

[0012] Figure 4 The image shows an IOL implanted in the eye according to certain embodiments and a Purkinje image of the reflectance that can be detected.

[0013] Figure 5 A system for estimating the offset of the IOL from the visual axis of the eye, according to certain embodiments, is shown.

[0014] Figure 6 This is a process flowchart of a method for evaluating the alignment of the IOL with the visual axis of the eye, according to certain embodiments.

[0015] Figure 7 An example computing device is shown that implements at least partially, according to certain embodiments, one or more functions for facilitating visualization during ophthalmic surgery.

[0016] For ease of understanding, the same reference numerals have been used where possible to refer to the same elements common to the figures. It is contemplated that elements and features of one embodiment can be advantageously combined in other embodiments without further description. Detailed Implementation

[0017] Figure 1 An example ophthalmic system 100 for performing ophthalmic treatments in a surgical setting is illustrated. System 100 includes an ophthalmic microscope 102. A surgeon 104 uses the ophthalmic microscope 102 to view structures on and within the eye 106 of a medical patient 108 within the field of view of the microscope 102. In this illustration, the ophthalmic microscope 102 is supported on an adjustable cantilever 110 of a microscope support base 112. The patient 108 can be supported on an operating table 114. The ophthalmic microscope 102 is movable in three dimensions with the cantilever 110, allowing the surgeon 104 to position the ophthalmic microscope 102 relative to the eye 106 of the patient 108 as needed.

[0018] In some embodiments, the ophthalmic microscope 102 includes a high-resolution, high-contrast stereoscopic surgical microscope. The ophthalmic microscope 102 will typically include a monocular eyepiece 116 or a binocular eyepiece 116 through which the surgeon 104 will obtain an optically magnified view of the relevant eye structures that the surgeon 104 will need to see to perform a given surgery or diagnose an eye condition in the patient 108.

[0019] The ophthalmic microscope 102 includes a digital camera and a broadband light source, a multispectral imaging (MSI) device, and / or other types of imaging devices for capturing color (red, green, and blue) images. The digital images captured using the camera can be displayed on a display device within the ophthalmic microscope 102.

[0020] The ophthalmic microscope 102 may include two display devices visible through binocular eyepieces 116 and displaying images of the patient's eyes 106 captured from different viewpoints by two cameras to provide stereoscopic observation. For example, the ophthalmic microscope 102 may be implemented as the NGENUITY 3D visualization system provided by Alcon Inc. in Fort Worth, Texas.

[0021] Images from the ophthalmic microscope 102 may be additionally or alternatively displayed on one or more display devices in the surgical environment. For example, one or more display devices may include a display device 118 that is fastened to a support arm 110 above the ophthalmic microscope 102.

[0022] To allow the surgeon 104 to obtain a stereoscopic view without constantly looking at the eyepiece 116, one or more display devices may further include a display device 120, which may be implemented as a three-dimensional display device. Thus, the display device 120 can provide a stereoscopic view of the image captured using the ophthalmic microscope 102. The display device 120 can be implemented as any type of three-dimensional display device known in the art, including three-dimensional display devices with or without special filter glasses. For some types of three-dimensional display devices, three-dimensional perception requires the viewer to be within a threshold distance from the display device 120. The display device 120 can be mounted on a cart, a manually adjustable arm or robotic arm, or other manually or automatically adjustable support.

[0023] Figure 2AThis is a schematic diagram of an ophthalmic microscope 102, which includes an input optics 200, a left illuminator optics 202a and a right illuminator optics 202b, a left microscope optics 204a and a right microscope optics 204b, and a left-eye camera 206a and a right-eye camera 206b. As used herein, "left" and "right" are used to refer to first and second instances of components for facilitating left and right-eye viewing for the surgeon 104. The use of "left" and "right" should be understood as exemplary only, and it should be understood that the two can be readily interchanged without altering their function.

[0024] The input optics 200 receives light reflected from the patient's eye 106. The input optics 200 may include a set of lenses having a common optical axis or two sets of lenses having offset and / or non-parallel optical axes (e.g., a right lens group and a left lens group). The left illuminator optics 202a and right illuminator optics 202b include light sources and optics that both (a) guide light from the light source to the eye 106 and (b) allow light reflected from the eye to pass through the illuminator optics 202a, 202b. Therefore, the left illuminator optics 202a and right illuminator optics 202b may each include a beam splitter and possibly one or more lenses to facilitate this function. The light sources of the left illuminator optics 202a and right illuminator optics 202b may be implemented as light-emitting diodes (LEDs) or other light sources. The light sources may operate at various intensities and colors. In some embodiments, the light source may include an LED having three different wavelength distributions (e.g., centered on red, green, and blue wavelengths) and independently selectable intensities, allowing control over the color emitted by the light source. The light source may additionally or alternatively include infrared or near-infrared light sources that enable one or both of the following: (a) illumination using light invisible to the patient; and (b) illumination using visible light that causes minimal discomfort to the patient.

[0025] Light reflected from eye 106 passes through left illuminator optics 202a and right illuminator optics 202b, and is magnified by left microscope optics 204a and right microscope optics 204b, respectively. The magnification of left microscope optics 204a and right microscope optics 204b can be adjusted. Similarly, the depth of focus of left microscope optics 204a and right microscope optics 204b can be adjusted. The light output reflected from eye is emitted by left microscope optics 204a and right microscope optics 204b to left camera 206a and right camera 206b. Left camera 206a and right camera 206b output images, which can be displayed on any of display devices 118, 120, the internal display of ophthalmic microscope 102, or other display devices. The images output by left camera 206a and right camera 206b can be further processed according to the methods described herein to evaluate the alignment of the IOL relative to eye 106.

[0026] In some embodiments, only one camera is used (referred to herein as a "single microscope camera"). In such embodiments, a single set of microscope optics and a single set of illuminator optics may be present. Alternatively, the illustrated pair of left microscope optics 204a and right microscope optics 204b, as well as left illuminator optics 202a and right illuminator optics 202b, may be used.

[0027] Light from illuminator optics 202a and 202b is transmitted to eye 106. The light is guided along the optical axes 210a and 210b of the left and right sides of the ophthalmic microscope 102 (i.e., the optical axes 210a and 210b of the left and right illuminator optics 202a and 202b, respectively). Optical axes 210a and 210b may be parallel to each other or converge at a point outward from the input optics 200. In some embodiments, additional illumination is provided by a paraxial light source 212 oriented along axis 214, which is not parallel to the optical axes 210a and 210b, for example, at an angle between 5 and 12 degrees. As used herein, light from the left and right illuminator optics 202a and 202b is referred to as “left coaxial light,” “right coaxial light,” or collectively “coaxial light.” Light from the paraxial light source 212 is referred to as “paraxial light.”

[0028] Figure 2B An example of how an ophthalmic microscope can be used to view a patient's eye 106 is shown. Light emitted by the left illuminator optics 202a and the right illuminator optics 202b is presented as two bright spots 216a, 216b, and light from the paraxial light source 212 is presented as a third bright spot 218 offset relative to the bright spots 216a, 216b.

[0029] Back Figure 2ALight from the left illuminator optics 202a and 202b, as well as the paraxial light source 212, is incident on the eye 106, and portions of this light are reflected from different surfaces of the eye 106. For example, a portion 220 of the light is reflected from the anterior surface of the cornea 222 and is referred to as the P1 Purkinje image. A portion is also reflected from the posterior surface of the cornea and is referred to as the P2 Purkinje image, but this is less noticeable and not commonly used. A portion 224 of the light is reflected from the anterior surface of the lens 226 and is referred to as the P3 Purkinje image. A portion 228 of the light is reflected from the posterior surface of the lens 226 and is referred to as the P4 Purkinje image. In practice, the P1 and P4 Purkinje images are the most obvious and commonly used clinically.

[0030] The controller 240 can be coupled to the left camera 206a and the right camera 206b to receive images output by the left camera 206a and the right camera 206b. The controller 240 can be implemented by the computing system 700 described below. The controller 240 can also be coupled to the left illuminator optics 202a and the right illuminator optics 202b and the paraxial light source 212 to control their operation.

[0031] refer to Figure 2C The IOL 242 can be implanted within the capsular bag 244 that previously housed the lens 226. The anterior surface of the capsular bag 244 and the lens 226 have been removed previously. A portion 246 of the light reflected from the anterior surface of the IOL 242 also produces the Purkinje image, sometimes referred to as the P1 Purkinje image of a pseudo-phakic eye (an eye with an artificial lens).

[0032] refer to Figure 3 IOL 242 can be implemented as the torus IOL shown. IOL 242 may include a lens portion 300, which includes a surface defining a lens. Lens portion 300 may be a refractive lens that depends on the curvature of the front and / or rear surfaces. Lens portion 300 may additionally or alternatively include diffractive elements, such as concentric refractive rings that provide multifocal vision.

[0033] The IOL 242 may include only the lens portion 300 and may include a peripheral portion 302 for engaging the pocket 244. Alternatively, the IOL 242 may include a loop 304 fastened to the peripheral portion. The loop 304 acts as a spring that pushes the pocket 244 outward to stabilize the IOL 242 and / or maintain the orientation of the IOL 242 within the pocket 244 about the visual axis 230. The lens portion 300 and / or the peripheral portion 302 may include markings 306 (e.g., reference markings) that can be used to determine the orientation of the IOL 242 about the visual axis 230 to facilitate astigmatism correction.

[0034] Figure 4Example images are shown that can be captured using the left camera 206a and the right camera 206b, or a single microscope camera. The images may show the sclera 400 and iris 402 of the eye 106. Both the sclera 400 and iris 402 may include distinguishing features that allow the images to be registered relative to known anatomical structures of the eye, such as enabling the images to be recorded in reference images as part of a treatment plan. The images further show the IOL 242, where marker 306 is visible. In some cases, portions of the loop 304 may also be visible. The IOL 242 is made of a transparent material, making the actual visibility much smaller than the visibility shown.

[0035] The image further includes one or more Purkinje images 404a, 404b. It is clear that the Purkinje images 404a, 404b can be flipped relative to each other, for example, since Purkinje image 404a is a reflection from the front surface of IOL 242 and Purkinje image 404b is a reflection from the rear surface of the pouch 244, that is, the reflection of light has passed through IOL 242 and is therefore flipped.

[0036] Figure 5 A system 500 is shown that can be implemented with respect to images received from the left camera 206a and the right camera 206b or a single microscope camera. In the following examples, a single time series of images is described as an "intraoperative image" from a "camera," which is understood to mean that a "camera" can be either the left camera 206a or the right camera 206b or a single microscope camera. When using the left camera 206a and the right camera 206b simultaneously, the results of system 500 processing intraoperative images at the same time step (e.g., captured within 10 milliseconds of each other) can be used in combination or individually, as indicated below.

[0037] System 500 includes machine learning model 502. Machine learning model 502 may include one or more of neural networks (NN), convolutional neural networks (CNN), or other types of artificial intelligence models. For example, machine learning model 502 may include one or more of the U-NET machine learning model, the You Only Look Once (YOLO) machine learning model, or another available machine learning model that can be trained to perform the tasks described herein as belonging to machine learning model 502.

[0038] Specifically, the machine learning model 502 can be trained to receive intraoperative images 504 and output segmented images 506. The segmented images 506 may include pixel labels 508 of the intraoperative images 504 corresponding to anatomical structures (e.g., sclera 400, iris 402, limbus or other anatomical structures), pixel labels 510 of the intraoperative images 504 corresponding to Purkinje images (e.g., Purkinje images 404a, 404b), and labels 512 of IOL features (e.g., loop 304, marker 306, edge of peripheral portion 302, edge of lenticule portion 300, diffraction ring, etc.).

[0039] Machine learning model 502 may include multiple machine learning models, each trained to generate labels of one type among labels 508, 510, and 512. In any case, machine learning model 502 may be trained using training data entries, each including training intraoperative images and one or more artificially generated labels, such as any one of labels 508, 510, and 512. Machine learning model 502 may process the training intraoperative images and generate one or more estimated labels, which may then be compared with one or more artificially generated labels, and machine learning model 502 may be updated based on the comparison.

[0040] A segmented image 506, which may include intraoperative images and one or more labels from labels 508, 510, and 512, may be processed by a visual axis registration module 514, which generates a visual axis overlay 518. The visual axis registration module 514 may further take one or more preoperative data 516 as input. For example, preoperative data 516 may include a reference image of the eye 106 and labels applied to the reference image, such as labels for visual axes, labels for one or more anatomical structures (e.g., any anatomical structures identified by label 58), and labels for axes defining the desired orientation of the IOL 242 (e.g., lines intersecting label 306). Preoperative data may include depth information, such as the depth of the posterior surface of the capsular bag 244 relative to the corneal apex, the depth of the iris plane, or the depth of other anatomical structures along the visual axis 230.

[0041] The visual axis registration module 514 estimates the position of the visual axis 230 in the intraoperative image 504. Since the intraoperative image is a two-dimensional image and the visual axis 230 is defined in three-dimensional space, the position of the visual axis determined by the visual axis registration module 514 can be an estimated point in the intraoperative image 504, which represents a surgeon-related point along the visual axis, such as a point of the capsular bag 244 along the visual axis (e.g., at a predetermined offset from the posterior surface of the capsular bag 244 and the corneal apex, as defined in the preoperative data).

[0042] For example, the label 510 of the Purkinje image can be used to determine the estimated position of the visual axis. If multiple Purkinje images are aligned in the intraoperative image 504, the visual axis can be estimated to be centered in the Purkinje images. If the Purkinje images are misaligned, the visual axis can be inferred to be at the midpoint of the line connecting the Purkinje images. Preoperative data 516 can be used to infer the position of the visual axis. For example, using a reference image included in the preoperative data 516, the position of the visual axis marked in the reference image can be mapped to the position on the preoperative image. Then, when the reference image is captured, the position of the Purkinje image (i.e., the magnitude and orientation of the misalignment) can be used to infer the tilt of the eye 106 relative to the ophthalmic microscope 102 and / or relative to the orientation of the eye 106. The visual axis determined using the reference image can then be adjusted in accordance with the inferred tilt to obtain the estimated visual axis. Adjusting the estimated visual axis with reference tilt can take into account depth information included in the preoperative data, such as the degree of translation based on tilt (angle) and depth information.

[0043] The processing result of the visual axis registration module 514 can be a visual axis overlay 518. The visual axis overlay 518 can be a label (e.g., pixel position) of the estimated position of the visual axis as defined above. The visual axis overlay 518 can be input to the alignment module 520. The alignment module 520 estimates the degree of misalignment between the estimated visual axis and the center of IOL 242. The alignment module 520 can identify the center of IOL 242, for example, by evaluating the label 512 of the IOL feature. For example, the alignment module 520 can identify the center of a pixel spot marked as corresponding to the lens portion 300 as the center of IOL 242. In another example, the alignment module 520 can determine the center of IOL 242 as the center of a pixel representing the combination of the lens portion 300 and the peripheral portion 302. In another example, the alignment module 520 can determine the center of IOL 242 as the center point between the labels 306. In yet another example, the alignment module 520 can also determine the center of IOL 242 by performing some other evaluation on the label 512.

[0044] The alignment module 520 can then output an axis offset 522, such as a representation of the difference between the estimated position of the view axis and the center of the IOL. This representation can be an offset (estimated X and Y offset distances), a graphical indication of the degree of misalignment (e.g., an arrow showing the direction in which the IOL should be moved to achieve alignment), or other representations. If the difference is below a threshold, the axis offset 522 can include an indication that no further alignment is required.

[0045] System 500 can be modified to incorporate additional functionality, wherein intraoperative images 504 (hereinafter referred to as "right intraoperative image and left intraoperative image") are simultaneously (e.g., within 10 milliseconds) received from left camera 206a and right camera 206b. For example, for each type of label 508, 510, 512, three-dimensional volumetric labels can be created using the labels obtained from the right and left intraoperative images. For example, stereoscopic vision techniques can be used to evaluate the label 510 of the Purkinje image in the left intraoperative image and the label 510 of the same Purkinje image in the right intraoperative image to obtain a three-dimensional representation of the Purkinje image, e.g., a volumetric image. Each anatomical structure item represented in label 508 and each IOL feature represented in label 512 can be processed in a similar manner to obtain a volumetric image. The three-dimensional location of the center of IOL 242 can be estimated based on the volumetric image obtained from the label 512 of the IOL feature. The three-dimensional location of the Purkinje image can be used to infer the location of the visual axis (or points along the visual axis as defined above) in three dimensions. The alignment can then be estimated by comparing the three-dimensional position of the view axis and the center of IOL 242 or some other reference point on IOL 242.

[0046] Figure 6 Method 600, which can be executed by controller 240 (e.g., controller 240 configured to implement system 500), is shown. Method 600 can be executed repeatedly, for example, for each frame (e.g., each time step) or every Nth frame (N is an integer greater than 1) of the video stream output by left camera 206a and right camera 206b or a single microscope camera.

[0047] In step 602, one or more intraoperative images are captured, and in step 604, each of the one or more intraoperative images is segmented using machine learning model 502 to obtain segmented images, such as segmented images including some or all of the labels 508, 510, 512 as defined above. Then, in step 606, the estimated position of the visual axis is determined based on the segmented intraoperative images, for example, using the methods described above regarding the functionality of the visual axis registration module 514. In step 608, the center of IOL 242 is determined based on the segmented images, and in step 610, the offset between the center of IOL 242 and the estimated position of the visual axis is determined, for example, the amount of interval between the center of IOL 242 and the estimated position of the visual axis. Steps 608 and 610 can be performed by implementing the functionality of the alignment module 520 as described above.

[0048] In step 612, the offset is evaluated relative to a threshold condition. If the threshold condition is not met, a representation of the offset is displayed in step 614. For example, step 614 may include any representation displaying the axis offset 522 as described above. The offset representation may be displayed on display device 120, a display device inside ophthalmic microscope 102, or some other display device. If the threshold condition is met, a success message may be displayed in step 616. The success message may be text, symbols, colors, or other visual attributes indicating that no further adjustment to IOL 242 is required. The threshold condition may be that the offset is below the maximum permissible offset.

[0049] For a given iteration of method 600 performing step 612, steps 612 and 614 or 616 can be performed substantially in real time, for example, within 10, 5, 4, 2 or 1 time steps of capturing intraoperative images in step 602.

[0050] Figure 7 An example computing system 700 is shown. The ophthalmic microscope 102 and the display device 120 can be combined with a computing device that has some or all of the attributes of the computing system 700.

[0051] As shown in the figure, the computing system 700 includes a central processing unit (CPU) 702, one or more I / O device interfaces 704 that allow various I / O devices 714 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to the computing system 700, a network interface 706 through which the computing system 700 is connected to a network 790, a memory 708, a storage device 710, and interconnects 712.

[0052] CPU 702 can retrieve and execute programming instructions stored in memory 708. Similarly, CPU 702 can retrieve and store application data residing in memory 708. Interconnect 712 transfers programming instructions and application data between CPU 702, I / O device interface 704, network interface 706, memory 708, and storage device 710. CPU 702 can be used to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc.

[0053] Memory 708 represents volatile memory (such as random access memory) and / or non-volatile memory (such as non-volatile random access memory, phase-change random access memory, etc.). As shown, memory 708 may store executable code for implementing one or both of controller 240 and system 500.

[0054] Storage device 710 may be a non-volatile memory, such as a disk drive, a solid-state drive, or a collection of storage devices distributed across multiple storage systems. Storage device 710 may optionally store preoperative data 516. Additional considerations

[0055] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. For example, changes can be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components can be suitably omitted, replaced, or added to various examples. Furthermore, features described with respect to some examples can be combined in some other examples. For example, any number of aspects set forth herein can be used to implement an apparatus or practice. Additionally, the scope of this disclosure is intended to cover such apparatus or methods practiced using other structures, functions, or structures and functions other than or different from the aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0056] As used herein, the phrase “at least one of a series of items” refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination of multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other order of a, b, and c).

[0057] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, operation, processing, derivation, investigation, searching (e.g., searching in a table, database, or other data structure), ascertainment, etc. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include parsing, selecting, picking, building, etc.

[0058] The methods disclosed herein include one or more steps or actions for implementing the methods. The method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Typically, where operations are illustrated in the figures, those operations may have corresponding means and functional components with similar numbering.

[0059] The various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration.

[0060] The processing system can be implemented using a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus can include any number of interconnect buses and bridges. The bus can link together various circuits, including processors, machine-readable media, and input / output devices. User interfaces (e.g., keypads, displays, mice, joysticks, etc.) can also be connected to the bus. The bus can also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and therefore will not be described further. The processor can be implemented using one or more general-purpose and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuit systems capable of executing software. Those skilled in the art will recognize how best to implement the described functions for the processing system, depending on the specific application and the overall design constraints imposed on the system as a whole.

[0061] If implemented in software, functionality can be stored or transmitted as one or more instructions or code on or through a computer-readable medium. Software should be interpreted broadly as instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or others. Computer-readable media includes both computer storage media and communication media (such as any medium that facilitates the transfer of computer programs from one place to another). The processor may be responsible for managing the bus and general processing, including the execution of software modules stored on the computer-readable storage medium. The computer-readable storage medium may be coupled to the processor, allowing the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. For example, the computer-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer-readable storage medium on which instructions separate from the wireless node are stored, all accessible to the processor via a bus interface. Alternatively or additionally, the computer-readable medium, or any portion thereof, may be integrated into the processor, for example, in cases where it may have a cache and / or a general-purpose register file. Examples of machine-readable storage media may include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, disks, optical disks, hard disks, or any other suitable storage media, or any combination thereof. Machine-readable media may be embodied in computer program products.

[0062] Software modules can comprise a single instruction or a number of instructions, and can be distributed across several different code segments, across different programs, and across multiple storage media. Computer-readable media can include multiple software modules. Software modules include instructions that, when executed by a device such as a processor, cause the processing system to perform various functions. Software modules can include transmission modules and reception modules. Each software module can reside in a single storage device or be distributed across multiple storage devices. For example, when a triggering event occurs, a software module can be loaded from a hard disk drive into RAM. During the execution of a software module, the processor can load some instructions into a cache to improve access speed. Then, one or more cache lines can be loaded into a general-purpose register file for processor execution. When referring to the functionality of a software module, it should be understood that this functionality is implemented by the processor when executing the instructions from that software module.

[0063] The following claims are not intended to be limited to the embodiments shown herein, but are given the full scope consistent with the language of the claims. In the claims, references to singular elements, unless specifically stated otherwise, are not intended to mean “one and only one”, but rather “one or more.” Unless otherwise specifically stated otherwise, the term “some” means one or more. No element of any claim shall be interpreted in accordance with 35 U.SC §112(f) unless such elements are expressly described using the phrase “means for…” or, in the case of a method claim, using the phrase “steps for…”. All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known to or will be known hereafter by one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended for public disclosure, whether or not such disclosure is expressly stated in the claims.

Claims

1. An ophthalmic visualization system, comprising: An ophthalmic microscope, the ophthalmic microscope including at least one camera configured to capture at least one image, the at least one image including a representation of a patient's eye located near the ophthalmic microscope; as well as A controller, coupled to the at least one camera, is configured to: Receive the at least one image from the at least one camera; The at least one image is segmented using a machine learning model to obtain at least one segmented image, the at least one segmented image including one or more labels of one or more Purkinje images represented in the at least one image; as well as Output is generated based on the at least one segmented image.

2. The ophthalmic visualization system as described in claim 1, wherein, The controller is further configured to: The estimated position of the visual axis of the patient's eye is calculated based on the one or more labels of the one or more Purkinje images; and The output is generated based on the estimated position of the line of sight.

3. The ophthalmic visualization system as described in claim 2, wherein, The controller is further configured to use preoperative data to calculate the estimated position of the visual axis of the eye.

4. The ophthalmic visualization system as described in claim 3, wherein, The preoperative data includes reference images of the patient's eyes.

5. The ophthalmic visualization system as described in claim 2, wherein, The at least one segmented image further includes one or more tags of an intraocular lens (IOL) located within the patient's eye, and the controller is further configured to: The estimated center of the IOL is calculated based on one or more labels of the IOL; Calculate the offset between the estimated center of the IOL and the estimated position of the line of sight; as well as The output is generated based on the offset.

6. The ophthalmic visualization system as described in claim 5, wherein, The controller is configured to: The offset is evaluated relative to a threshold condition; and If the offset meets the threshold condition, the output is generated as a success message.

7. The ophthalmic visualization system as described in claim 5, wherein, The controller is configured to: The offset is evaluated relative to a threshold condition; and If the offset does not meet the threshold condition, the output is generated as a representation of the offset.

8. The ophthalmic visualization system as described in claim 1, wherein, The machine learning model includes at least one of a neural network (NN) or a convolutional neural network (CNN).

9. The ophthalmic visualization system as described in claim 1, wherein, The machine learning model includes at least one of U-NET or You Only See Once (YOLO) machine learning models.

10. The ophthalmic visualization system as described in claim 1, wherein, The at least one camera includes a left camera and a right camera, and the at least one image includes at least two images captured substantially simultaneously by the left camera and the right camera.

11. A method for ophthalmic visualization, comprising: The controller receives at least one image from at least one camera of an ophthalmic microscope, with the patient's eye in the field of view of the ophthalmic microscope; The controller uses a machine learning model to segment the at least one image to obtain at least one segmented image, the at least one segmented image including one or more labels of one or more Purkinje images represented in the at least one image; as well as The controller generates an output based on the at least one segmented image.

12. The method of claim 11, further comprising: The controller calculates the estimated position of the visual axis of the patient's eye based on the one or more labels of the one or more Purkinje images; as well as The controller generates the output based on the estimated position of the line of sight.

13. The method of claim 12, further comprising using preoperative data to calculate an estimated position of the visual axis of the eye by the controller.

14. The method of claim 13, wherein, The preoperative data includes reference images of the patient's eyes.

15. The method of claim 12, wherein, The at least one segmented image further includes one or more tags of an intraocular lens (IOL) located within the patient's eye, and the method further includes: The controller calculates the estimated center of the IOL based on one or more labels of the IOL; The controller calculates the offset between the estimated center of the IOL and the estimated position of the line of sight; and The controller generates an output based on the offset.

16. The method of claim 15, further comprising: The offset is evaluated by the controller relative to a threshold condition; The controller determines whether the offset satisfies the threshold condition. as well as In response to determining that the offset meets the threshold condition, the controller outputs a success message.

17. The method of claim 15, further comprising: The offset is evaluated by the controller relative to a threshold condition; as well as The controller determines whether the offset does not meet the threshold condition. In response to determining that the offset does not meet the threshold condition, the output is generated as a representation of the offset.

18. The method of claim 11, wherein, The machine learning model includes at least one of a neural network (NN) or a convolutional neural network (CNN).

19. The method of claim 11, wherein, The machine learning model includes at least one of the U-NET machine learning model or the You Only See Once (YOLO) machine learning model.

20. The method of claim 11, wherein, The at least one camera includes a left camera and a right camera, and the at least one image includes at least two images captured substantially simultaneously by the left camera and the right camera.

Images