Systems and methods for registration of multimodal imaging systems incorporated insurgical microscopes

Abstract

Described herein are systems and methods for registration of multimodal imaging systems incorporated in surgical microscopes. For example, there is a multimodal imaging system, including: one or more object-space telecentric digital microscope channels configured to generate data for detecting anatomical landmarks in a surgical field; and one or more object-space optical coherence tomography (OCT) channels configured to perform a scan based on data from the one or more object-space telecentric digital microscope channels including the identified anatomical landmarks; in which the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels share a coaxial optical axis. Other examples are described.

Description

SYSTEMS AND METHODS FOR REGISTRATION OF MULTIMODAL IMAGING SYSTEMS INCORPORATED IN SURGICAL MICROSCOPESCLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Application No. 63 / 684,301, filed August 16, 2024, the contents of which are incorporated herein by reference in their entirety.INCORPORATION BY REFERENCE

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.FIELD

[0003] The present invention relates to surgical microscopes employing multiple optical imaging modalities (multimodal imaging systems) and, more particularly, to ophthalmic surgical microscopes employing both digital microscopy and optical coherence tomography.BACKGROUND

[0004] Ophthalmic surgical microscopes traditionally have been “analog”, i.e. the surgeon’s retinas function as the ultimate “detectors” of light returning from the sample, and thus the surgeon’s eyes are part of the optical system of the microscope. There has recently been a push to develop “digital” microscopes, i.e. microscopes that use complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) sensors to detect light returning from the sample. This has numerous advantages and some disadvantages, which have been described in detail elsewhere.

[0005] Briefly, the advantages of digital microscopes include that they are more sensitive to light than the surgeon’ s retina, and are compatible with digital gain, and thus can reduce the amount of light the surgeon needs to use, reducing phototoxicity. They can also have higher resolution than a human retina, particularly off-axis, and this higher resolution can be conveyed to the surgeon via magnification through a digital display, either a 3D TV or head-mounted display. Finally, they also remove the surgeon’s anterior eye from the microscope equation, eliminating problems associated with accommodation (in younger surgeons) and aberrations from the surgeon’ s own anterior eye, ultimately resulting in higher image quality.- I of 27 -SG Docket No.: 14843-709.600

[0006] Digital microscopes also have a key advantage for the adoption of newer technologies, as they provide access to the surgical images in digital format. Traditionally, recording of surgical procedures has been achieved through camera channels that are either connected to one of the microscope ocular channels through a beamsplitter, or are in a completely different channel. In either case, as these channels are typically only used for recording, they are not always the highest quality and do not contain stereoscopic (stereo) information, which may include a 3D image based on creating an illusion of depth from two flat images. Digital microscopes allow recording or transmitting of both stereo channels of the surgical microscope, in high quality and at full resolution.

[0007] Another recent advancement in ophthalmic surgical microscopes is the integration of optical coherence tomography (OCT) directly into the optical train of surgical microscopes. This integration confers numerous advantages, and some workflow and usability challenges, which have been described in detail elsewhere.

[0008] Briefly, integrating OCT into a surgical microscope provides the surgeon with real-time information about the underlying tissue microstructure, and how it responds to surgical maneuvers. This information is fundamentally different from the enface, or top-down, surface view that is provided by the traditional microscope, and can provide vital information in all types of ophthalmic surgeries. In addition, OCT systems can provide quantitative depth information, as the OCT datasets are depth-resolved and three-dimensional. However, OCT datasets may suffer from errors due to distortion inherent to the imaging optics, and refraction in the subject eye. Both of these effects can be numerically corrected, but this requires additional processing. While intra-surgical OCT offers surgeons a new dimension of information, a fully three- dimensional dataset of the entire surgical field takes much longer to acquire than the two- dimensional datasets of a traditional microscope. Thus, even for the fastest OCT systems, OCT acquisition necessarily has slower refresh rates and longer latencies than surgical microscope video.

[0009] Because OCT volume acquisition rates are relatively slow, they cannot self-contextualize on time scales relevant to ophthalmic surgery. In other words, although a full volumetric OCT dataset contains enough information to identify key surgical landmarks (such as surgical tools and major anatomical features), the rate at which full volumetric OCT datasets can be acquired, at least with current OCT technologies, is not fast enough to provide a surgeon with feedback on a reasonable timescale to facilitate performing ophthalmic surgery using OCT data alone. As a result, intraoperative OCT systems operate simultaneously with traditional surgical microscopes,- 2 of 27 -SG Docket No.: 14843-709.600relying on some form of digital recording of surgical microscope video and then registering the OCT volume to the surgical microscope images.

[0010] The registration of OCT data to surgical microscope images is complex and dynamic, leading to several practical challenges. To provide valuable quantitative depth information, microscope-integrated OCT systems are usually designed to be object-space telecentric. That is, the OCT system is designed such that the chief rays emanating from the microscope’ s common main objective are substantially parallel to the optical axis of the microscope’s common main objective, such that the lateral magnification of the OCT volume does not change as a function of depth. Furthermore, many microscope-integrated OCT systems are designed with the OCT system centered on the clear aperture of the common main objective and with an optical axis that is coaxial with the common main objective optical axis.

[0011] While object-space telecentricity is all that is required for consistent registration between telecentric and OCT, this does require that the entire microscope move up and down to control the focus of the microscope. Otherwise, if the focus were adjusted by moving the image sensors, the image would remain telecentric at a given focus setting, but the magnification would vary with changing focus. Note that this does not affect the OCT, which can adjust its focus through the use of an adjustable collimator (in some embodiments, a liquid lens) without affecting magnification. One potential solution to this problem is to make the telecentric channel both object space telecentric and image space telecentric. In such a configuration, the focus could be adjusted by translating the camera sensor (e.g. CCD or CMOS ) without moving the entire microscope, and the magnification would remain unchanged.

[0012] Contrastingly, the microscope ocular channels are designed to provide the surgeon with depth perception through stereopsis. As a result, they are decidedly not telecentric, as doing so would eliminate the depth cues that come from changes in lateral magnification with depth. Furthermore, the ocular channels are typically arranged to be decentered on the common main objective, and angularly offset from its optical axis, to provide the surgeon with depth perception through stereopsis.

[0013] A typical combined stereoscopic microscope and telecentric OCT system 102 is depicted diagrammatically in FIGS. 1A-1B. The resulting angular offset between the telecentric, centered OCT channel 104 and the non-telecentric, non-coaxial digital microscope (DM) channel 106 (including left microscope channel 105 and right microscope channel 107) cause significant challenges for registration. While registration at a single plane is possible, the microscope channels’ angular offset and variable magnification make registration depth-dependent (for example when the combined system is zoomed into a region between a common main objective- 3 of 27 -SG Docket No.: 14843-709.600and image plane 108), significantly increasing the complexity of three-dimensional registration. OCT channel 104 and digital microscope channel 106 are also shown as a zoomed region between the common main objective and image plane.

[0014] Turning to FIG. 1A, telecentric OCT system 102 includes OCT fold mirror 120, OCT beam expansion optics 122, stereo DM fold mirrors 124, stereo DM CCD 126, stereo DM imaging optics 128, coaxial illumination beamsplitter 130, dichroic mirror 132, common main objective 134, and object / image plane 140.

[0015] Image plane 108 includes common main objective 136 and object / image plane 140.

[0016] In certain examples, centered OCT channel 104, left microscope channel 105, and right microscope channel 107 pass through the combined system 102 and image plane 108.

[0017] FIG. IB shows a view of non-telecentric, non-coaxial digital microscope (DM) channel 106, through which left 105 and right 107 microscope channels pass. Also shown is common main objective 136 and object / image plane 140. A pathway 104A for telecentric, centered OCT channel 104 is also shown, including common main objective 136.

[0018] The traditional configuration shown in FIGS. 1A-1B lacks a centered camera channel, and thus the lateral position of each pixel varies as a function of depth. Thus, any calibration with the OCT is necessarily depth-dependent, requiring first order corrections for position before even applying more advanced corrections such as those for distortion and depth dependent magnification.

[0019] It should further be noted that simply adding a centered microscope channel would not solve these problems.

[0020] FIGS. 1C-1D depicts such a system.

[0021] FIG. 1C includes a view 112 in which a centered microscope channel 110 is added for registration with a telecentric OCT channel 104 (shown here as zoomed into a region between the common main objective and image plane). Shown is OCT fold mirror 120, OCT beam expansion optics 122, centered DM fold mirror 138, centered DM CCD 139, centered DM imaging optics 129, coaxial illumination beamsplitter 130, dichroic mirror 132, common main objective 134, and object / image plane 140.

[0022] However, because the microscope channel is not telecentric, it still suffers from depthdependent magnification, and thus the registration between the OCT and microscope channel remains complex and depth-dependent (as shown in view 118 showing the combined system zoomed into a region between the common main objective 136 and object / image plane 140).- 4 of 27 -SG Docket No.: 14843-709.600

[0023] FIG. ID shows another view 104B of telecentric, centered OCT channel 104, which passes through common main objective 136. Also shown is a view 110A of centered microscope channel 110 passing through common main objective 136 towards object / image plane 140.

[0024] Thus, configurations with centered but non- telecentric cameras, which are not common in surgical microscopes but may be used in other applications obviate the need for "first order" position corrections, because they can be calibrated at the image plane and the calibration may remain valid for small displacements from the image plane. However, they still suffer from depth dependent magnification and depth dependent distortion, meaning for objects even moderately above or below the image plane, co-registration with the OCT fails quickly.

[0025] Furthermore, any changes to the optical train of the system (for example, the additional of a wide angle viewing system or even the contribution of a patient’s eye’s cornea and lens) will differentially affect the OCT and microscope channels, invalidating any pre-existing registration and making a new registration challenging, if not impossible, to perform.

[0026] As such, there remains a need for improved systems and procedures for combining OCT and microscope channels that do not experience disruptions with depth-dependent magnification and allow for ease of registration between the OCT and microscope channel, which may be implemented for efficiently scanning and mapping anatomical landmarks as part of surgical procedures.SUMMARY OF THE DISCLOSUREDescribed herein is a multimodal imaging system, including: one or more object-space telecentric digital microscope channels configured to generate data for detecting anatomical landmarks in a surgical field; and one or more object-space optical coherence tomography (OCT) channels configured to perform a scan based on data from the one or more object-space telecentric digital microscope channels including the identified anatomical landmarks; in which the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels share a coaxial optical axis.

[0027] Tn various aspects, the system further includes the shared coaxial optical axis of the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels being one or more of coaxial with an optical axis of a common main objective and approximately centered on a clear aperture of the common main objective.

[0028] In various aspects, light emanating from the common main objective is configured to be substantially parallel to the optical axis to provide a constant magnification as a function of depth.- 5 of 27 -SG Docket No.: 14843-709.600

[0029] In various aspects, the common main objective is configured to combine the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels with other channels of the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels.

[0030] In various aspects, the one or more object-space telecentric digital microscope channels are two object-space telecentric digital microscope channels configured to provide stereopsis.

[0031] In various aspects, the one or more object-space telecentric digital microscope channels are three object-space telecentric digital microscope channels, wherein two of the three channels are configured to provide stereopsis and one of the three channels is configured to be telecentric.

[0032] In various aspects, the coaxial optical axis of least one pair of the object-space telecentric digital microscope channels and object-space telecentric OCT channels is non-perpendicular to an object plane.

[0033] In various aspects, the one or more object-space digital microscope channels and the one or more object-space telecentric OCT channels are combined using a dichroic mirror.

[0034] In yet other aspects, the dichroic mirror is positioned in an infinity space above a common main objective.

[0035] In various aspects, the multimodal imaging system is configured for performing one or more of microsurgery and robotic-assisted surgery via interfacing with a robotic system, wherein the surgical field includes an eye of a patient.

[0036] In various aspects, each one of the one or more object-space telecentric digital microscope channels has a corresponding one of the one or more object-space optical coherence tomography (OCT) channels.

[0037] In various aspects, the system employs a scan pattern of one or more of:(i) 3D / volume scans, (ii) line scans including Raster, cross-sectional and parallel line scans, (iii) radial scans including K scan patterns (iv) special scans including grid scans, mesh scans and C- scans; further in which the scan pattern is based on procedural requirements including resolution, speed and refresh rate.

[0038] In various aspects, the one or more object-space telecentric digital microscope channels are image-space telecentric, further in which a focus of the one or more object-space telecentric digital microscope channels is configured to be adjusted by translating an image sensor, in which the image sensor is one or more of a charge coupled device (CCD) and complementary metal oxide semiconductor (CMOS) detector, in which one or more digital microscopes producing the object-space telecentric digital microscope channels remain stationary.- 6 of 27 -SG Docket No.: 14843-709.600

[0039] In still other aspects, there is provided a method of scanning an eye for performing a surgical procedure using a multimodal imaging system. First, there is a step of generating data from one or more object-space telecentric digital microscope channels. Next, there is a step of processing the generated data from the one or more object-space telecentric digital microscope channels to detect anatomical landmarks in a surgical field. A next step includes translating coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels into an optical coherence tomography (OCT) scan pattern coordinates. Still further, there is a step of acquiring OCT data from one or more object-space telecentric OCT channels. The next step in the method includes performing a scan based on the OCT scan pattern coordinates vie the one or more object-space telecentric OCT channels to detect the anatomical landmarks. Still next, there is the step of registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan. A final step involves transferring one or more of the detected anatomical landmarks from the one or more object-space telecentric digital microscopes channels and the OCT scan to a coordinate frame of a robot system. In certain examples, anatomical landmarks may be generic or non-anatomical landmarks. In certain examples, the scan or scan pattern may be a cross-sectional scan.

[0040] Telecentricity is important for automated systems and robotic surgery in order to equate pixels and OCT registration to avoid distortion due to distance and depth across different imaging modalities.

[0041] In certain examples, detecting the anatomical landmarks includes producing labels of ocular structures.

[0042] In yet other examples, registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan includes use of deep learning models using the one or more object-space telecentric digital microscope channels and object-space telecentric OCT channels as inputs.

[0043] In still other examples, performing the scan based on the OCT scan pattern coordinates provides rapid real-time visual guidance and monitoring of the surgical field during a surgical procedure.

[0044] In still other examples, registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan is based on common landmark locations detected by both the one or more object-space telecentric digital microscope channels and the one or more object-space telecentric OCT channels.- 7 of 27 -SG Docket No.: 14843-709.600

[0045] In still other examples, the robot system plans and executes a trajectory path for the surgical procedure based on the transferred detected anatomical landmarks to the coordinate frame of the robot system.

[0046] In still other examples, the detected anatomical landmarks include components of the robot system including a surgical tool tip.

[0047] In still other variations, the method includes combining anatomical landmarks detected by the one or more object-space telecentric digital microscope channels and the one or more object-space telecentric OCT channels to produce three-dimensional features.

[0048] In still other variations, the method includes determining spatial relationships between one or more of the coordinate frames of the robot system, the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels, and coordinates from the OCT scan.

[0049] In still other variations, the method further includes transferring data from one or more of: the coordinate frame of the robot system, the one or more object-space telecentric digital microscope channels, and the one or more object-space telecentric OCT channels to other software modules.

[0050] Advantageously, the registration of OCT and microscope imaging modalities is able to be used efficiently in autonomous or semi- autonomous robot- assisted surgical systems. The motion of a surgical robot to perform autonomous surgery is calculated based on relevant anatomical landmark locations in three-dimensional space. The anatomical landmarks are detected in both RGB and OCT images and transformed to the robot’ s three-dimensional coordinate frame to be used in the tool path generation using registration between subsystems. The OCT volume scan provides enough data to detect the key landmarks. However, the slow acquisition time of a full volume scan is a limiting factor to be used in guiding the surgical robot in real-time. In visual guidance and monitoring of the surgical robot, digital microscope images are utilized to understand the two-dimensional enlace view of the surgery in addition to color information. Then an OCT scan is acquired by aiming it using the subset of landmarks detected in a digital microscope image to obtain information of the eye.

[0051] The OCT scan is acquired significantly faster than a volume scan and often provides sufficient information of the eye to be relevant for control of the surgical robot. Combination of the telecentric digital microscope system and OCT allows an accurate mapping from the landmarks detected in digital microscope image to the scan pattern of OCT and vice versa, which is infeasible when the stereo system is used because of challenges described previously. This- 8 of 27 -SG Docket No.: 14843-709.600fusion of digital microscope image and OCT B-scan enables the fast and sufficient local data acquisition and thus the real-time visual guidance and monitoring.

[0052] All and each of the methods and apparatuses described herein, in any combination, are herein contemplated and can be used to achieve the benefits as described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0053] A better understanding of the features and advantages of the methods and apparatuses described herein will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings of which:

[0054] FIGS. 1A-1B depicts a traditional integration of a stereoscopic digital microscope (DM) and telecentric OCT.

[0055] FIGS. 1C-1D depicts a centered but non-telecentric DM channel combined with telecentric OCT channel.

[0056] FIGS. 2A-2B depicts a centered telecentric DM combined with a telecentric OCT.

[0057] FIGS. 2C-2D depicts another illustrative embodiment of an aspect of the multimodal imaging system wherein a telecentric OCT is combined with a three channel DM.

[0058] FIG. 2E depicts the combined telecentric OCT with the three channel DM.

[0059] FIG. 3 depicts techniques in tissue segmentation using multimodal imaging.

[0060] FIG. 4 depicts registration between imaging modalities.

[0061] FIG. 5 depicts OCT scan patterns being generated from the data originating in the telecentric microscope channel of multimodal imaging.

[0062] FIG. 6 depicts three-dimensional anatomical features generated by combining multimodal imaging data.

[0063] FIGS. 7A-7B is a flow chart for a method of scanning an eye for performing a surgical procedure using a multimodal imaging system.DETAILED DESCRIPTION

[0064] FIGS. 2A-2B depict the inventive concept of a centered telecentric DM combined with a telecentric OCT.

[0065] FIG. 2A shows a view 202 of the combined centered telecentric DM with telecentric OCT. Shown are OCT 204 and DM 207 channels. Also shown is a view 208 of the combined system zoomed into a region between the common main objective 236 and the object / image plane. Components of this combined centered telecentric DM and telecentric OCT system include OCT fold mirror 220, telecentric DM fold mirror 242, telecentric DM CCD 244,- 9 of 27 -SG Docket No.: 14843-709.600telecentric DM imaging optics 228, dichroic mirror 232, common main objective 234 and object / image plane 240.

[0066] FIG. 2B shows a view 204A of OCT channel 204 passing through common main objective 236, and a view 207A of DM channel 207 passing through common main objective 236 and towards object / image plane 240.Integration with Autonomous Surgical Robotic Systems

[0067] The registration of OCT and microscope imaging modalities is advantageous when used in autonomous or semi-autonomous robot-assisted surgical systems. The motion of a surgical robot to perform autonomous surgery is calculated based on relevant anatomical landmark locations in three-dimensional space. The anatomical landmarks are detected in both RGB and OCT images and transformed to the robot’s three-dimensional coordinate frame to be used in the tool path generation using registration between subsystems. The OCT volume scan provides enough data to detect the key landmarks. However, the slow acquisition time of a full volume scan is a limiting factor to be used in guiding the surgical robot in real-time. In visual guidance and monitoring of the surgical robot, digital microscope images are utilized to understand the two-dimensional enface view of the surgery in addition to color information. Then an OCT scan is acquired by aiming it using the subset of landmarks detected in a digital microscope image to obtain information of the eye. The OCT scan is acquired significantly faster than a volume scan and often provides sufficient information of the eye to be relevant for control of the surgical robot. Combination of the telecentric digital microscope system and OCT allows an accurate mapping from the landmarks detected in digital microscope image to the scan pattern of OCT and vice versa, which is infeasible when the stereo system is used because of challenges described previously. This fusion of digital microscope image and OCT B-scan enables fast and sufficient local data acquisition and thus the real-time visual guidance and monitoring.

[0068] The object of this invention is to provide an approach to designing and constructing an optical microscope with an integrated OCT such that the registration between the OCT channel and at least one channel of the microscope is fast, accurate, and valid across the entire imaging depth OCT system, and remains valid under certain system perturbations.

[0069] While the techniques disclosed herein are described for ophthalmic surgical applications, they are equally applicable to other surgical applications, non-surgical and non-medical applications, and to other combinations of imaging modalities (for any application) where a two- dimensional imaging modality is used as the basis of registration for a three-dimensional modality.- 10 of 27 -SG Docket No.: 14843-709.600

[0070] The invention comprises at least one microscope channel that is object-space telecentric, and at least one OCT channel that is both object-space telecentric and coaxial with the telecentric microscope channel. Because both channels are telecentric and coaxial, a single two-dimensional registration of the two channels remains valid over a wide depth range. Furthermore, the introduction of additional optics (such as a wide-angle viewing system or even the anterior segment of a subject eye) affects both the telecentric microscope channel and the OCT channel equally (disregarding minor effects of chromatic aberration), obviating the need for dynamic correction of the registration.

[0071] For widefield (as opposed to point-scanning) imaging systems, object-space telecentricity can be realized by any optical arrangement that places the aperture stop (or, in some cases, an image of the aperture stop) a focal length behind the front group or objective, such that the entrance pupil is projected to infinity. For a microscope with a common main objective, this requires that an aperture stop (or an image of the aperture stop) be placed at the back focal plane of the common main objective.

[0072] For point scanning imaging systems, such as OCT systems, object-space telecentricity can be realized by placing the scanning optics, or an image of the scanning optics, a focal length behind the front group or objective, such that the chief rays emerging from the front group or objective are substantially parallel to each other for all positions of the scanning optics. For a microscope with a common main objective, this requires that the scanning mirrors (or an image of the scanning mirrors) be placed at the back focal plane of the common main objective.

[0073] The inclusion of a telecentric channel may be motivated by a need for co-registration with OCT. Since microscope-integrated OCT systems are often designed to be telecentric, in order to achieve a simplified, depth-independent co-registration with a camera, the camera also needs to be telecentric. In traditional surgery done by humans, intraoperative OCT may be used, and such co-registration is not critical since the OCT is self-contextualizing and may provide an OCT-data derived enface view. However, co-registering OCT becomes important in the context of automation / machine vision systems. By co-registering OCT and telecentric systems, the machine vision system may save computational time and resources if the telecentric and OCT are well-registered. One beneficial result is the inventive configuration can also initially analyze telecentric camera information, which may update much more quickly and require less processing than volumetric OCT covering the same field. The more rapid analysis of telecentric camera information may then be used to determine whether and where to update OCT information. Thus, in contrast to imagining systems with human operators, the problems of- 11 of 27 -SG Docket No.: 14843-709.600computational time and contextualization become important considerations with the integration of machine vision and robotics into ophthalmic surgery.

[0074] Still further, adaptation of telecentricity in a surgical microscopy system may require significant changes to the setup of microscope optics, including requiring moving the imaging optics far behind the objective lens. For the system to telecentric, the imaging optics need to be located a distance behind the objective lens that is equal to the sum of the effective focal length of the objective lens and the imaging optics, such that the entrance pupil is projected to infinity.

[0075] Thus, arranging the optics to accommodate telecentricity makes the imaging system larger, as can be seen in the difference between the location of the fold mirrors such as centered DM fold mirror 138 in FIGS. 1C as compared to telecentric DM fold mirror 242 in FIG. 2 A. Thus, telecentricity may necessitate optical systems with lens separations that are much larger than normally seen in traditional surgical imaging systems.

[0076] In some embodiments, the scanning optics may be a single reflecting surface, two- dimensional scanning mirror such as a fast-steering mirror, piezo scanning mirror or MEMS mirror. In such cases it is trivial to place the mirror, or an image of it, in the back focal plane of the common main objective.

[0077] In some embodiments, the scanning optics may comprise a pair of galvanometers, or some combination of single axis scanning mirrors including galvanometers, resonant scanners, polygonal scanners or similar devices. In such cases, telecentricity may be achieved by imaging the mirrors onto each other, and either placing one of the mirrors, or an image of both mirrors, in the back focal plane of the objective or front group. Alternatively, the scanning mirrors can be placed in close axial proximity, and then either placed near the back focal plane of the front group or objective or be imaged near the back focal plane of the front group or objective. In such embodiments, telecentricity in one or both axes may be imperfect, resulting in a small amount of fan distortion. Depending on the application, residual fan distortion may be acceptable, as it may be small for small mirror offset-distances and can be corrected or compensated for in software.

[0078] In some embodiments, the telecentric microscope channel may have a fixed magnification (no optical zoom), leveraging high pixel count CMOS or CCD sensors and appropriate optical design to achieve the requisite resolution and field-of-view (FOV). In such embodiments, a variable aperture may be used to independently adjust the numerical aperture (or, equivalently, the f-number or depth-of-field) of the telecentric channel, to simulate the effects of an optical zoom system.

[0079] In some embodiments, the telecentric microscope channel may be doubly telecentric, enabling focusing of the telecentric microscope channel via translation of the image sensor,- 12 of 27 -SG Docket No.: 14843-709.600though this is not required. Focus control in this manner will not affect the telecentricity of the telecentric microscope channel, and thus any registration remains valid during focus adjustment.

[0080] In some embodiments, the OCT channel may facilitate focus control through the use of a variable collimator, which may in turn make use of a translating lens or deformable (liquid) lens, though this is not required. Focus control in this manner will not affect the telecentricity of the OCT channel, and thus any registration remains valid during focus adjustment.

[0081] In some embodiments, both the telecentric microscope channel and the OCT channel would be nominally centered on the clear aperture of a common main objective, and / or have their central chief rays being coaxial with the optical axis of the common main objective.

[0082] In some embodiments, the telecentric microscope channel and OCT channel may be combined, through a shared common main objective, with additional microscope channels. These additional microscope channels may or may not be telecentric, and may be used for depth perception, via stereopsis, photogrammetry, or related techniques.

[0083] In some embodiments, the telecentric microscope channel and the OCT channel are neither centered on the clear aperture of a common main objective, nor coaxial with the optical axis of a common main objective.

[0084] In some embodiments, the telecentric microscope channel may be one of two or more channels used for depth perception, via stereopsis, photogrammetry, or related techniques.

[0085] In some embodiments, there may be more than one telecentric microscope channel, and each telecentric microscope channel may have its own corresponding coaxial OCT system.

[0086] In certain embodiments, object-space telecentricity is sufficient to achieve consistent spatial registration between the telecentric imaging channel and the OCT channel. However, under such conditions, it is generally necessary to translate the entire microscope assembly along the optical axis in order to adjust the focus of the imaging system. If, instead, the focus were to be adjusted by translating the image sensor (e.g., a CCD or CMOS detector), the system would maintain telecentricity at a single focal setting, but the effective magnification would vary as a function of focus depth. This variation in magnification arises because image-space telecentricity is not preserved during sensor translation in a purely object-space telecentric configuration.

[0087] Notably, this limitation does not affect the OCT subsystem, which can incorporate an adjustable collimation mechanism to achieve dynamic focusing without altering magnification. In some embodiments, this mechanism includes a tunable collimator or a variable-focus optical element, such as a liquid lens.

[0088] To address the aforementioned limitation, the telecentric imaging channel may be- 13 of 27 -SG Docket No.: 14843-709.600configured to be both object-space and image-space telecentric. In such a configuration, axial translation of the image sensor may be used to adjust focus without requiring movement of the entire microscope assembly. Because both entrance and exit pupils are effectively positioned at infinity in a doubly telecentric system, magnification remains substantially invariant with focus adjustments.

[0089] FIGS. 2C-2D depict another illustrative embodiment of an aspect of the multimodal imaging system wherein a telecentric OCT 204 is combined with a three channel DM 212, comprising both a stereo DM channel 211 having mono DM channels 205 and 207 and a centered telecentric DM channel 213.

[0090] FIG. 2C shows a view 204A of telecentric OCT channel 204 passing through common main objective 236.

[0091] FIG. 2D shows a view of a three channel DM 212, which includes a view 211 A of mono DM channels 205 and 207, as well as a view 213A of mono DM channel 205, each passing through common main objective 236 towards object / image plane 240.

[0092] FIG. 2E depicts the combined telecentric OCT 284 with the three channel DM 281 / 282 / 283. Telecentric OCT 281 may start from an intermediate image plane, and three channel DM may include a telecentric DM 283 and a stereo DM 281 / 282.

[0093] Various channels passing through such a system include telecentric OCT 204, stereo DM 211, centered telecentric DM 213, and three channel DM 212.

[0094] Components of this combined system 284 / 281 / 282 / 283 include OCT fold mirrors 246, OCT beam expansion optics 248, OCT fold mirror 250, OCT beam expansion optics 252, left stereo CCD 256A, telecentric CCD 260, stereo fold mirror left 258B, stereo fold mirror right 258A, stereo CCD right 256A, stereo CCD left 256B, OCT fold mirror 262, DM imaging optics 264, coaxial illumination beamsplitter 266, dichroic mirror 268, common main objective 270 and objective plane 240.Multimodal Tissue Segmentation

[0095] In some embodiments, the telecentric microscope channel registered with the OCT channel can be processed through a multimodal deep learning model that takes both channels as input, process them through a series of mathematical operations (including convolution and maxpooling) to produce labels of ocular structures such as the cornea, iris, and pupil. The combination of the two modalities provides mutually beneficial constraints that reduce the number of training images necessary to train the models. The redundant information is also central to provide robustness to outliers and make more accurate predictions in the presence of occlusions that might occur within one of the modalities. While multimodal deep learning- 14 of 27 -SG Docket No.: 14843-709.600models have proven effective in many areas, they have not been explored in the context of ophthalmic surgery.

[0096] FIG. 3 depicts techniques in tissue segmentation using multimodal imaging. Shown here are top-down views of an eye 304 and 306 during a surgical procedure taken by a telecentric digital microscope 302, for example along an OCT target axis b which may correspond to a vertical axis x and a horizontal axis y. Surgical instruments such as a probe, cutter and retractor are also shown in 304 and 306. Also shown are corresponding scans 354 and 356 of views 304 and 306 taken as OCT cross-sections 352 in real-time, which may correspond to a horizontal axis b and a vertical axis z.Trajectory Generation Based on Multimodal Data

[0097] The coordinate frames of OCT and telecentric microscope channel data can be registered using common landmark locations detected in both modalities. The coordinate frame of the robotic system can be registered to the coordinate frame of the OCT using data that represents the same landmark positions represented in the robot and OCT frames. This landmark position can be a mechanical part of the robot system (e.g., tool tip) and the same part can be detected and described in OCT frame using manual or automatic detection of it. With these registration data, the spatial relationship between coordinate frames of three modalities (OCT, telecentric microscope, and robotic system) can be known, and the data in any of those modalities can be transferred to other software modules. One such example is the anatomical landmark detection results in which both imaging modalities can be transferred to the robot coordinate frame and used for trajectory generation for autonomous operation of surgery. Another example is that some parts of a robotic system can be described in imaging coordinate frames for providing the prior information to computer vision or Al algorithms in images to ease and enhance the detection problem. Lastly, but not limited to, landmark detection results in one imaging modality can be transformed to another imaging modality coordinate to provide prior information.

[0098] FIG. 4 depicts registration between imaging modalities. Shown here is a field of view for OCT 424 and telecentric digital microscope 410, as well as registration 499 between the imaging system 402 and a robotic system 403.OCT Targeting via DM / AI

[0099] In some embodiments, OCT scan patterns may be generated from the data originating in the telecentric microscope channel. The data of the microscope channel is first processed to detect anatomical features, through traditional image-processing, artificial-intelligence, or other such means. The coordinates of these detected features in the microscope channel data can be- 15 of 27 -SG Docket No.: 14843-709.600translated into OCT scan pattern coordinates of any arbitrary pattern. The generated scan pattern can be used to acquire OCT data at the coordinates specified in the scan pattern.

[0100] FIG. 5 depicts OCT scan patterns 598 being generated from the data originating in the telecentric microscope channel 510 of multimodal imaging system 502, which may also include the OCT system 524. Telecentric microscope channel 510 may generate feature identification 597 which may detect and, in some examples, label anatomical features 596, which may then be sent to an OCT targeter 525 to generate an OCT scan pattern 595 to be sent to the OCT system 524.

[0101] FIG. 6 depicts three-dimensional anatomical features 654 generated by combining multimodal imaging data such as a top (two-dimensional) view taken by a telecentric digital microscope 602 and an OCT cross-sectional view 652.

[0102] Anatomical features detected in the OCT channel data can be combined with features detected in the telecentric microscope channel data to produce features in three dimensions. Microscope channel data is two-dimensional but can be higher resolution in the lateral directions than OCT data. OCT data, in contrast, can be three-dimensional, but has highest resolution in its axial (depth) direction. By using a subset of dimensions from one imaging modality and combining them with a subset of dimensions from the other imaging modality, the combined result can exhibit benefits of each.

[0103] FIGS. 7A-7B is a flow chart for a method of scanning an eye for performing a surgical procedure using a multimodal imaging system 700-701.

[0104] Method 700-701 begins at block 705 of FIG. 7A with generating data from one or more object-space telecentric digital microscope channels.

[0105] Method 700-701 continues at block 710 with processing the generated data from the one or more object-space telecentric digital microscope channels to detect anatomical landmarks in a surgical field.

[0106] Next, at block 715, method 700-701 includes translating coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels into an optical coherence tomography (OCT) scan pattern coordinates.

[0107] Method 700-701 continues at block 720 with acquiring OCT data from one or more object-space telecentric OCT channels. In certain examples, real-time OCT scanning by the multimodal imaging system, including sub-selecting a portion of the eye or tool tip area may employ various scan patterns, for example depending on whether resolution or speed / refresh rate is preferred, such as higher resolution for polishing procedures and higher speed / refresh rates for capsulotomy. Exemplary scan patterns may include:- 16 of 27 -SG Docket No.: 14843-709.600

[0108] 3D or volume scans,

[0109] line scans including Raster or parallel line scans to provide cross-sectional views, cross-line scans, line density scans,

[0110] radial scans, including K scan patterns, and

[0111] special scans including grid scans, mesh scans, and C-scans (en face images) that sum data from B-scans.

[0112] Method 700-701 continues at block 725 of FIG. 7B with performing a scan based on the OCT scan pattern coordinates vie the one or more object-space telecentric OCT channels to detect the anatomical landmarks.

[0113] Next, method 700-701 continues at block 730 with registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan.

[0114] Finally, at block 735, method 700-701 concludes with transferring one or more of the detected anatomical landmarks from the one or more object-space telecentric digital microscopes channels and the OCT scan to a coordinate frame of a robot system.

[0115] According to certain examples of method 700-701, detecting the anatomical landmarks includes producing labels of ocular structures.

[0116] According to certain examples of method 700-701, registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan includes use of deep learning models using the one or more object-space telecentric digital microscope channels and object-space telecentric OCT channels as inputs.

[0117] According to certain examples of method 700-701, performing the scan based on the OCT scan pattern coordinates provides rapid real-time visual guidance and monitoring of the surgical field during a surgical procedure.

[0118] According to certain examples of method 700-701, registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan is based on common landmark locations detected by both the one or more object-space telecentric digital microscope channels and the one or more object-space telecentric OCT channels.

[0119] According to certain examples of method 700-701, method 700-701 further includes the robot system planning and executing a trajectory path for the surgical procedure based on the transferred detected anatomical landmarks to the coordinate frame of the robot system.- 17 of 27 -SG Docket No.: 14843-709.600

[0120] According to certain examples of method 700-701, the detected anatomical landmarks include components of the robot system including a surgical tool tip.

[0121] According to certain examples of method 700-701, method 700-701 further includes combining anatomical landmarks detected by the one or more object-space telecentric digital microscope channels and the one or more object-space telecentric OCT channels to produce three-dimensional features.

[0122] According to certain examples of method 700-701, method 700-701 further includes determining spatial relationships between one or more of the coordinate frame of the robot system, the coordinates of the detected anatomical landmarks from the one or more objectspace telecentric digital microscope channels, and coordinates from the OCT scan.

[0123] According to certain examples of method 700-701, method 700-701 further includes transferring data from one or more of: the coordinate frame of the robot system, the one or more object-space telecentric digital microscope channels, and the one or more object-space telecentric OCT channels to other software modules.

[0124] According to certain examples of method 700-701 , method 700-701 further includes adjusting a focus of one or more of the object-space telecentric digital microscope channels via translating an image sensor, and further maintaining stationary one or more digital microscopes producing the object-space telecentric digital microscope channels.

[0125] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein and may be used to achieve the benefits described herein.

[0126] The process parameters and sequence of steps described and / or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0127] Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency,- 18 of 27 -SG Docket No.: 14843-709.600intensity, etc.), determining, alerting, or the like. For example, any of the methods described herein may be performed, at least in part, by an apparatus including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the processes(s) of the method.

[0128] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0129] The various exemplary methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any method as disclosed herein can be combined with any one or more steps of any other method as disclosed herein.

[0130] When a feature or element is herein referred to as being "on" another feature or element, it can be directly on the other feature or element or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0131] Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or- 19 of 27 -SG Docket No.: 14843-709.600"comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0132] Spatially relative terms, such as "under", "below", "lower", "over", "upper" and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures is inverted, elements described as "under" or "beneath" other elements or features would then be oriented "over" the other elements or features. Thus, the exemplary term "under" can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms "upwardly", "downwardly", "vertical", "horizontal" and the like are used herein for the purpose of explanation only unless specifically indicated otherwise.

[0133] Although the terms “first” and “second” may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Thus, a first feature / element discussed below could be termed a second feature / element, and similarly, a second feature / element discussed below could be termed a first feature / element without departing from the teachings of the present invention.

[0134] Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0135] In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and / or steps may alternatively be exclusive, and may be expressed as “consisting of’ or alternatively “consisting essentially of’ the various components, steps, sub-components or sub-steps.- 20 of 27 -SG Docket No.: 14843-709.600

[0136] As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word "about" or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and / or position to indicate that the value and / or position described is within a reasonable expected range of values and / or positions. For example, a numeric value may have a value that is + / - 0. 1% of the stated value (or range of values), + / - 1% of the stated value (or range of values), + / - 2% of the stated value (or range of values), + / - 5% of the stated value (or range of values), + / - 10% of the stated value (or range of values), etc. Any numerical values given herein should also be understood to include about or approximately that value, unless the context indicates otherwise. For example, if the value "10" is disclosed, then "about 10" is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that "less than or equal to" the value, "greater than or equal to the value" and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value "X" is disclosed the "less than or equal to X" as well as "greater than or equal to X" (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0137] Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.

[0138] The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and- 21 of 27 -SG Docket No.: 14843-709.600logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.- 22 of 27 -SG Docket No.: 14843-709.600

Claims

CLAIMSWhat is claimed is:

1. A multimodal imaging system, comprising: one or more object-space telecentric digital microscope channels configured to generate data for detecting anatomical landmarks in a surgical field; and one or more object-space optical coherence tomography (OCT) channels configured to perform a scan based on data from the one or more object-space telecentric digital microscope channels including the identified anatomical landmarks; wherein the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels share a coaxial optical axis.

2. The system of claim 1, further comprising the shared coaxial optical axis of the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels being one or more of coaxial with an optical axis of a common main objective and approximately centered on a clear aperture of the common main objective.

3. The system of claim 2, wherein light emanating from the common main objective is configured to be substantially parallel to the optical axis to provide a constant magnification as a function of depth.

4. The system of claim 3, wherein the common main objective is configured to combine the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels with others channels of the one or more object-space OCT channels and the one or more object-space telecentric digital microscope channels.

5. The system of claim 1, wherein the one or more object-space telecentric digital microscope channels are two object-space telecentric digital microscope channels configured to provide stereopsis.

6. The system of claim 1, wherein the one or more object-space telecentric digital microscope channels are three object-space telecentric digital microscope channels, wherein two of the three channels are configured to provide stereopsis and one of the three channels is configured to be telecentric.- 23 of 27 -SG Docket No.: 14843-709.6007. The system of claim 1 , wherein the coaxial optical axis of least one pair of the objectspace telecentric digital microscope channels and object-space telecentric OCT channels is nonperpendicular to an object plane.

8. The system of claim 1 , wherein the one or more object-space digital microscope channels and the one or more object-space telecentric OCT channels are combined using a dichroic mirror.

9. The system of claim 8, wherein the dichroic mirror is positioned in an infinity space above a common main objective.

10. The system of claim 1, wherein the multimodal imaging system is configured for performing one or more of microsurgery and robotic-assisted surgery via interfacing with a robotic system, wherein the surgical field includes an eye of a patient.

11. The system of claim 1, wherein each one of the one or more object-space telecentric digital microscope channels has a corresponding one of the one or more object-space optical coherence tomography (OCT) channels.

12. The system of claim 1, wherein the system, via the OCT channels, employs a scan pattern of one or more of: (i) 3D / volume scans, (ii) line scans including Raster and parallel line scans, (iii) radial scans including K scan patterns and (iv) special scans including grid scans, mesh scans and C-scans; further wherein the scan pattern is based on procedural requirements including resolution, speed and refresh rate.

13. The system of claim 1, wherein the one or more object-space telecentric digital microscope channels are image-space telecentric, further wherein a focus of the one or more object-space telecentric digital microscope channels is configured to be adjusted by translating an image sensor, wherein the image sensor is one or more of a charge coupled device (CCD) and complementary metal oxide semiconductor (CMOS) detector, wherein one or more digital microscopes producing the object-space telecentric digital microscope channels remain stationary.- 24 of 27 -SG Docket No.: 14843-709.60014. A method of scanning an eye for performing a surgical procedure using a multimodal imaging system, comprising: generating data from one or more object-space telecentric digital microscope channels; processing the generated data from the one or more object-space telecentric digital microscope channels to detect anatomical landmarks in a surgical field; translating coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels into optical coherence tomography (OCT) scan pattern coordinates; acquiring OCT data from one or more object- space telecentric OCT channels; performing a scan based on the OCT scan pattern coordinates via the one or more objectspace telecentric OCT channels to detect the anatomical landmarks; registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan; and transferring one or more of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan to a coordinate frame of a robot system.

15. The method of claim 14, wherein detecting the anatomical landmarks includes producing labels of ocular structures.

16. The method of claim 14, wherein registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan includes use of deep learning models using the one or more object-space telecentric digital microscope channels and object-space telecentric OCT channels as inputs.

17. The method of claim 14, wherein performing the scan based on the OCT scan pattern coordinates provides rapid real-time visual guidance and monitoring of the surgical field during a surgical procedure.

18. The method of claim 14, wherein registering the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels and the OCT scan is based on common landmark locations detected by both the one or more objectspace telecentric digital microscope channels and the one or more object-space telecentric OCT channels.- 25 of 27 -SG Docket No.: 14843-709.60019. The method of claim 14, further comprising the robot system planning and executing a trajectory path for the surgical procedure based on the transferred detected anatomical landmarks to the coordinate frame of the robot system.

20. The method of claim 14, wherein the detected anatomical landmarks include components of the robot system including a surgical tool tip.

21. The method of claim 14, further comprising combining anatomical landmarks detected by the one or more object-space telecentric digital microscope channels and the one or more object-space telecentric OCT channels to produce three-dimensional features.

22. The method of claim 14, further comprising determining spatial relationships between one or more of the coordinate frame of the robot system, the coordinates of the detected anatomical landmarks from the one or more object-space telecentric digital microscope channels, and coordinates from the OCT scan.23 . The method of claim 14, further comprising transferring data from one or more of: the coordinate frame of the robot system, the one or more object-space telecentric digital microscope channels, and the one or more object-space telecentric OCT channels to other software modules.

24. The method of claim 14, further comprising adjusting a focus of the one or more of the object-space telecentric digital microscope channels via translating an image sensor, further comprising maintaining stationary one or more digital microscopes producing the object-space telecentric digital microscope channels.- 26 of 27 -SG Docket No.: 14843-709.600

Images