Robotic Imaging System Using Orbital Scanning Mode

Abstract

The robotic imaging system includes a stereoscopic camera configured to record left and right images of a target site. A robotic arm is operatively connected to the stereoscopic camera, the robotic arm adapted to selectively move the stereoscopic camera relative to the target. The stereoscopic camera includes an optical assembly having at least one lens and defining a working distance. The optical assembly has at least one focus motor adapted to move the at least one lens to selectively vary the working distance. The robotic imaging system includes a controller having a processor and a tangible, non-transitory memory having instructions recorded thereon. The controller is adapted to selectively execute an orbital scanning mode to cause the robotic arm to sweep an orbital trajectory at least partially circumferentially around the eye while maintaining focus.

Description

[Technical field]

[0001] The present disclosure generally relates to a robotic imaging system. More particularly, the present disclosure relates to an orbital scanning mode in a robotic imaging system. Various imaging modalities, such as stereoscopic microscopes, are commonly used to image different parts of the human body. Some surgeries may require moving a camera over multiple areas in real time to obtain focused images. Conventional microscopes do not have the capability to perform this operation. In addition, it is difficult to provide a minimally invasive imaging technique to scan over delicate and small areas of the human body. Summary of the Invention [Means for solving the problem]

[0002] Disclosed herein is a robotic imaging system for imaging a target site. The robotic imaging system comprises a stereoscopic camera configured to record left and right images of a target site to generate at least one stereoscopic image of the target site. A robotic arm is operatively connected to the stereoscopic camera, the robotic arm adapted to selectively move the stereoscopic camera relative to the target. The stereoscopic camera comprises an optical assembly having at least one lens and defining a working distance. The optical assembly has at least one focus motor adapted to move the at least one lens to selectively vary the working distance. A controller is in communication with the stereoscopic camera and has a processor and a tangible, non-transitory memory having instructions recorded thereon.

[0003] The controller is adapted to selectively execute an orbital scanning mode to cause the robotic arm to sweep an orbital trajectory at least partially circumferentially around the eye while maintaining focus. The controller may be configured to determine a change in target depth from an initial target position, the change in target depth being defined as a displacement of a position of the target site along an axial direction. The controller is configured to update the particular focal length based in part on the change in target depth. The target site may include the ora serrata of the eye.

[0004] The orbital trajectory may be defined in spherical coordinate axes to define a first spherical angle and a second spherical angle. The controller is adapted to vary a viewing angle of the orbital trajectory by holding the first spherical angle constant while iterating the second spherical angle until a desired viewing angle is reached. The controller is adapted to selectively direct the orbital trajectory by iterating the first spherical angle between a predefined start angle and a predefined end angle while holding the second spherical angle constant at the desired viewing angle. In some embodiments, the orbital trajectory forms at least partially a circle. In some embodiments, the orbital trajectory forms at least partially an ellipse. The orbital trajectory may range in angle from about 180 degrees to 300 degrees. The orbital trajectory may range in angle from about 360 degrees.

[0005] The controller may be configured to center the stereoscopic cameras on the ocular reference surface and estimate a first working distance relative to the ocular reference surface. The controller may be adapted to change the view vector of the stereoscopic cameras to a desired viewing angle. The controller may be configured to lock a respective position of each target point along the orbital trajectory by constraining a respective position of the stereoscopic cameras to an outer surface of a virtual sphere, the virtual sphere defining a radius equal to a particular focal length. The particular focal length may be based in part on the desired viewing angle, the eye dimensions, and the first working distance.

[0006] The controller may be configured to determine a change in height of the stereoscopic camera from an initial camera position, the change in height being defined as a displacement of the position of the stereoscopic camera along an axial direction. The controller may be configured to update a particular focal length based in part on the change in height of the stereoscopic camera.

[0007] When the robot arm is no longer moving, the controller may be configured to determine a motor command for at least one focus motor that corresponds to a maximum sharpness position. The maximum sharpness position is based on one or more sharpness parameters including the sharpness signal, the maximum sharpness signal, and a time derivative of the maximum sharpness. At each update cycle, the controller may be configured to inject a respective difference value into each coordinate position of the orbital trajectory.

[0008] Disclosed herein is a stereoscopic imaging system for imaging a target site in an eye. The stereoscopic imaging system comprises a stereoscopic camera configured to record left and right images of the target site to generate at least one stereoscopic image of the target site. A robotic arm is operatively connected to the stereoscopic camera, the robotic arm adapted to selectively move the stereoscopic camera relative to the target site. The stereoscopic camera comprises an optical assembly having at least one lens and defining a working distance, the optical assembly having at least one focus motor adapted to move the at least one lens to selectively vary the working distance. A controller is in communication with the robotic arm and has a processor and a tangible, non-transitory memory having instructions recorded thereon. The controller is adapted to selectively execute an orbital scanning mode to cause the robotic arm to sweep an orbital track at least partially circumferentially around the eye while maintaining focus. The target site may include the ora serrata of the eye.

[0009] The orbital trajectory may be defined in spherical coordinate axes to define a first spherical angle and a second spherical angle. The controller is adapted to vary the viewing angle of the orbital trajectory by holding the first spherical angle constant while iterating the second spherical angle until a desired viewing angle is reached. The controller is adapted to selectively direct the orbital trajectory by iterating the first spherical angle between a predefined start angle and a predefined end angle while holding the second spherical angle constant at the desired viewing angle. The sharpness signal may be defined as the contrast between respective edges of the object in the at least one stereoscopic image. The maximum sharpness signal may be defined as the maximum sharpness value observed during the scanning period.

[0010] The above and other features and advantages of the present disclosure will become readily apparent from the following detailed description of the best mode for carrying out the disclosure when read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a partial schematic diagram of a robotic imaging system having a stereoscopic camera and a controller using an orbital scanning mode. [Diagram 2] 2 is a partial schematic diagram of exemplary optical components of the stereoscopic camera of FIG. 1. [Diagram 3] FIG. 1 is a schematic cross-sectional view of a portion of an eye. [Figure 4] FIG. 4 is a schematic partial cross-sectional view of the eye taken through axis 4-4 of FIG. 3. [Diagram 5] FIG. 2 is a schematic diagram of a virtual sphere that can be used in the robotic imaging system of FIG. 1. [Figure 6] 2 is a flow chart of an exemplary method for operating the orbital scanning mode of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Exemplary embodiments of the present disclosure are shown in the drawings as non-limiting examples and will be described in more detail below. It should be understood, however, that the novel aspects of the present disclosure are not limited to the specific forms shown in the drawings listed above. Rather, the present disclosure encompasses modifications, equivalents, combinations, subcombinations, permutations, groupings, and alternatives that are included within the scope of the present disclosure, such as those encompassed by the appended claims.

[0013] Referring to the drawings, in which like reference numbers refer to like components, FIG. 1 illustrates generally a robotic imaging system 10 having a stereoscopic camera 12 employing an orbital scanning mode 14. The robotic imaging system 10 is configured to image a target site 16. The orbital scanning mode 14 allows the surgeon to view parts of the eye without actually touching the eye, thereby avoiding contact procedures such as scleral compression. This allows for a quicker examination of the eye, shortens surgical time, and reduces the possibility of trauma.

[0014] 1, stereoscopic camera 12 is disposed at least partially within head unit 18 of housing assembly 20, which is configured to be pointed at least partially toward target site 16. Stereoscopic camera 12 is configured to record first and second images of target site 16, which images may be used to generate a live two-dimensional stereoscopic view of target site 16. Target site 16 may be an anatomical location on a patient, a biological specimen in a laboratory, a calibration slide / template, or the like.

[0015] With reference to FIG. 1, at least one selector 22 may be attached to the head unit 18 to select a particular feature of the stereoscopic camera 12, such as magnification, focus, and other features. The selector 22 may be utilized to allow an operator to manually position the head unit 18. The robotic imaging system 10 may include a robot arm 24 operatively connected to the head unit 18 and configured to selectively move the head unit 18. With reference to FIG. 1, the head unit 18 may be mechanically coupled to the robot arm 24 via a coupling plate 26. An operator may position and orient the stereoscopic camera 12 with assistance from the robot arm 24. A sensor 28 may be operatively connected to the robot arm 24 and / or the coupling plate 26. The sensor 28 may be configured to detect forces and / or torques applied by an operator to move the stereoscopic camera 12. The controller C may be adapted to calculate a trajectory for moving the robot arm 24 in a spherical manner while maintaining a constant target viewpoint.

[0016] The robot arm 24 may include one or more joints, such as a first joint 30 and a second joint 32, configured to provide an additional degree of positioning and / or orientation of the head unit 18. Data from the sensor 28 may be used to determine which joint of the robot arm 24 should rotate and how quickly the joint should rotate to provide assisted movement of the stereoscopic camera 12 corresponding to the force / torque provided by the operator. Referring to FIG. 1, each joint may be coupled to a respective joint motor (e.g., joint motor 31) and a respective joint sensor (e.g., joint sensor 33). The joint motor 31 is configured to rotate the first joint 30 about an axis, while the joint sensor 33 is configured to transmit the position of the first joint 30 (in 3D space).

[0017] 1, a robotic imaging system 10 includes a controller C having at least one processor P and at least one memory M (or non-transitory tangible computer readable storage medium) having instructions recorded therein for executing one or more subroutines and methods, such as a method 400 for operating an orbital scanning mode 14 (described with respect to FIG. 6). The memory M can store a set of controller executable instructions, and the processor P can execute the set of controller executable instructions stored in the memory M. The method 400 provides a workflow, robotic operation, and focus motor adjustment for moving the stereoscopic camera 12 while keeping the image in focus at all times, without the need to physically move the eye 200 (see FIG. 2). Conventional microscopes do not have the capability to perform this operation.

[0018] With reference to FIG. 1, the robot arm 24 may be controlled via an embedded processor, such as a controller C and / or a robot arm controller 42. The robot arm 24 may be selectively operable to extend the field of view of the stereoscopic camera 12 along the X-axis, the Y-axis, and the Z-axis. The head unit 18 may be connected to a cart 34 having at least one display medium, such as first and second displays 36 and 38 shown in FIG. 1 (which may be a monitor, a terminal, or other forms of two-dimensional visualization). With reference to FIG. 1, the controller C may be configured to process signals for broadcast on the first and second displays 36 and 38. The housing assembly 20 may be completely self-contained and movable between various locations. The image streams from the stereoscopic camera 12 may be sent to the controller C and / or a camera processor (not shown), which may be configured to prepare the image streams for viewing. For example, the controller C may combine or interleave the first and second video signals from the stereoscopic camera 12 to create a stereoscopic signal. The controller C may be configured to store the video and / or stereoscopic video signals in a video file and stored in the memory M. The first and second displays 36 and 38 may incorporate a stereoscopic display system, with a two-dimensional display having separate images for the left and right eyes. To view a stereoscopic display, a user may wear special glasses that operate in conjunction with the first and second displays 36, 38 to present a left view to the user's left eye and a right view to the user's right eye.

[0019] 1, the first display 36 may be connected to the cart 34 via a flexible mechanical arm 40 having one or more joints to allow for flexible positioning. The flexible mechanical arm 40 may be configured to have a length sufficient to extend over the patient to provide a relatively close view to the surgeon during surgery. The first and second displays 36 and 38 may include any type of display, such as a high definition television, an ultra high definition television, smart glasses, a projector, one or more computer screens, a laptop computer, a tablet computer, and / or a smartphone, and may include a touch screen.

[0020] The stereoscopic camera 12 is configured to acquire stereoscopic images of the target region 16, which may be presented in a variety of forms, including, but not limited to, captured still images, real-time images, and / or digital video signals. As used herein, "real-time" generally refers to updating information at the same rate as the data is received. More specifically, "real-time" means that image data is acquired, processed, and transmitted at a sufficiently high data rate and sufficiently low latency so that when the data is displayed, the object moves smoothly without any jerks or wait times that are noticeable to the user. Typically, real-time occurs when new images are acquired, processed, and transmitted at a rate of at least about 30 frames per second (fps) and displayed at about 60 fps, and when the combined processing of the video signals has a latency of about 1 / 30th of a second or less.

[0021] I. Optical components 2, an exemplary layout of the optical components of the stereoscopic camera 12 is presented. It should be understood that other optical components or devices available to one of ordinary skill in the art may be used. Images from the target site 16 are received by the stereoscopic camera 12 via an optical assembly 102 shown in FIG. 2. The optical assembly 102 includes a front lens 104 and a rear lens 106 within a housing 108.

[0022] 2, a focal plane 122 is located at a distance equal to a particular focal distance F from a principal plane 124 of the optical assembly 102. Visualization of objects with the stereoscopic cameras 12 above and below the focal plane 122 will cause the objects to be in focus. Because it can be difficult to measure the location of the principal plane 124, the distance from the bottom surface of the housing 108 to the focal plane 122 is designated as the working distance W. The working distance W may accurately set the plane or scene in focus of the target site 16.

[0023] The optical assembly 102 is configured to provide various working distances W (see FIG. 1) to the stereoscopic camera 12. Referring to FIG. 2, the controller C is adapted to selectively direct the focus motor 110 to vary the spacing between the rear lens 106 and the front lens 104. The focus motor 110 is movable (e.g., along a direction 112) to vary the working distance W of the optical assembly 102. As mentioned above, the working distance W may be referred to as the distance from the stereoscopic camera 12 to a reference plane on which the target site 16 is focused. In some embodiments, the working distance W is the distance from the optical origin of the stereoscopic camera 12 to a predefined reference plane on the target site 16. In some embodiments, the working distance W is adjustable from 200 to 450 mm by moving the rear lens 106 via the focus motor 110.

[0024] The relative movement of the rear lens 106 with respect to the front lens 104 also changes the particular focal length F, which is bounded by the maximum and minimum working distances allowed by the hardware. The focal length may be defined as the distance between the rear lens 106 and the front lens 104 plus one-half the thickness of the front lens 104. In one example, the focus motor 110 is an electric motor. However, the focus motor 110 may be any type of linear actuator, such as a stepper motor, a shape memory alloy actuator, or other type of actuator available to one skilled in the art.

[0025] In the illustrated embodiment, the rear lens 106 is movable along a direction 114 (here the Z-axis) while the front lens 104 is stationary. However, it should be understood that the front lens 104 may be movable or both the front lens 104 and the rear lens 106 may be movable. The focus motor 110 may be selectively operable via the controller C and / or the motor controller 116. The stereoscopic camera 12 may include an additional focus motor to move the front and rear lenses independently. The orientation of the stereoscopic camera 12 is indicated by a view vector 118 in 3D space.

[0026] In some embodiments, the front lens 104 is comprised of a plano-convex lens and / or a meniscus lens. The rear lens 106 may include an achromatic lens. In examples where the optical assembly 102 includes an achromatic refractive assembly, the front lens 104 may include a hemispherical lens and / or a meniscus lens. The rear lens 106 may include an achromatic doublet lens, a group of achromatic doublet lenses, and / or an achromatic triplet lens. The optical assembly 102 may include other types of refractive or reflective assemblies and components available to one of ordinary skill in the art. The magnification of the optical assembly 102 may vary based on the working distance W. For example, the optical assembly 102 may have a magnification of 8.9x for a working distance of 200mm and a magnification of 8.75x for a working distance of 450mm.

[0027] With reference to FIG. 2, imaging an object at focal plane 122 forms a conjugate image located at infinity behind or behind optical assembly 102. Optical assembly 102 is configured to provide left and right views of target site 16 via optical device 130. In some embodiments, optical device 130 comprises left and right optical units 132, 134 having respective sensors and optical devices. As shown in FIG. 2, left and right optical units 132, 134 generate left and right optical paths 136 and 138, respectively, which are two parallel optical paths within housing 108. Left and right optical units 132, 134 are laterally separated by distance 140. In some embodiments, distance 140 is about 58-70 mm. Outside the housing 108, the left and right optical paths 136, 138 extend to left and right optical axes 142, 144, respectively, in a direction slightly different than the optical axis 146 of the optical assembly 102. The left and right optical axes 142, 144 coincide at the center of the field of view at a target point 148. The target point 148 is sometimes referred to as the “tip” of the stereoscopic camera 12 at the focal plane 122.

[0028] Adjusting the relative positions of the front lens 104 and the rear lens 106 creates a new working distance WB at the new focal plane 122B. Referring to Figure 2, movement of the rear lens 106 causes a realignment of the left optical axis 142B and the right optical axis 144B, resulting in a relocated tip 148B of the stereoscopic camera 12.

[0029] The front lens 104 and the rear lens 106 are configured together to provide an infinite conjugate image to provide optimal focus for a downstream optical image sensor. In other words, an object located exactly at the focal plane of the target site 16 will have its image projected at an infinite distance, thereby providing infinity conjugate at the working distance provided. In general, an object will appear in focus at a certain distance from the focal plane along the optical path. However, past a certain threshold distance, the object will begin to appear blurry and out of focus.

[0030] 2 provides images of the target site 16 for both a left optical path 136 and a right optical path 138. In an alternative embodiment, the optical assembly 102 may include separate left and separate right front lenses 104 and separate left and separate right rear lenses 106. Additionally, each of the rear lenses 106 may be independently adjustable.

[0031] II. Orbital Scanning Mode 1, the controller C is adapted to selectively execute an orbital scanning mode 14 for the stereoscopic camera 12, which may be selectively initiated by a user. The orbital scanning mode 14 leverages two available images (left and right) of the stereoscopic camera 12 to execute an orbital trajectory and maintain image focus without the need for tedious manual focusing.

[0032] In some embodiments, the orbital scanning mode 14 begins when an operator selects the orbital scanning mode 14 (e.g., via an input device 66 such as a mouse (see FIG. 1 )), which causes a command message or signal to be sent to the controller C and / or the robot arm controller 42. Once the command is received, the example controller C and / or the robot arm controller 42 may record the current working distance, magnification, focus, and / or other optical parameters of the stereoscopic camera 12. The current position of the target site 16 may be determined from position data of the various joints of the robot arm 24. The controller C and / or the robot arm controller 42 may also record a current image of the target site 16.

[0033] The orbital scanning mode 14 may also be used in any stereoscopic visualization device with a variable (e.g., changeable) working distance W. FIG. 1 shows the working distance W, which may be defined as the distance between a reference plane and the focal plane of the target site 16. Thus, the working distance W sets the plane or in-focus scene of the target site 16. In the example shown in FIG. 2, the reference plane is the outer surface of the front lens (not shown) in the optical assembly 102. The working distance W may correspond to the angular field of view, with a longer working distance resulting in a wider field of view or a larger visible area. Based in part on the robot arm input and dynamic image feedback, the orbital scanning mode 14 automatically adjusts the working distance W to achieve improved image quality in various situations.

[0034] FIG. 3 is a partial schematic cross-sectional view of an eye 200 having a pupil 204 and a retina 206. FIG. 4 is a partial schematic cross-sectional view taken through axis 4-4 of FIG. 3. With reference to FIGS. 3-4, the eye 200 includes a lens 208, a sclera 210, and a choroid 212. The sensory region of the eye 200 begins at the ora serrata 214. In the embodiment shown in FIGS. 3-4, the target site 16 (see FIG. 1) includes or covers the ora serrata 214 of the eye 200. However, it should be understood that other portions of the eye 200 or the human body may be imaged. The ora serrata 214 is a sawtooth-like junction between the retina 206 and the corona ciliaris 216 (see FIG. 4, also referred to as the ciliary crown), and defines the transition point between the non-sensory region of the eye 200 and the multi-layered sensory region.

[0035] The robotic imaging system 10 provides the surgeon with a way to perform an orbital scan of the eye 200, for example, starting from and / or looking at the ora serrata 214 (see FIGS. 3-4). This allows the surgeon to quickly look for abnormalities such as retinal tears, peripheral breaks, residual vitreous, etc. The controller is programmed to move the stereoscopic camera 12 without physically moving the eye 200 while constantly focusing the image.

[0036] As described below, an operator / surgeon can position the robotic imaging system 10 about the eye 200 and engage the orbital scanning mode 14, and the robotic imaging system 10 can move to view the ora serrata 214 of the eye 200. Once the ora serrata 214 is reached, the surgeon can begin moving the robotic arm 24 in an orbital trajectory 230 that scans circumferentially around the eye 200. Multiple orbital trajectories 230 may be performed.

[0037] The controller C may be adapted to provide an application programming interface (API) for starting and stopping each orbital trajectory of the orbital scanning mode 14. The shape of the orbital trajectory 230 may be modified based on the application at hand. In some embodiments, the orbital trajectory 230 forms at least a partial circle. In other embodiments, the orbital trajectory 230 forms at least a partial ellipse. With reference to FIG. 4, the orbital trajectory 230 may subtend an angle 232 of approximately 180 degrees to 300 degrees. In some embodiments, the angle 232 is approximately 350 degrees. The orbital trajectory 230 may include multiple 360 ​​degree rotations, assuming the robotic arm 24 has sufficient joint limit clearance. Additionally, the orbital trajectory 230 may either stop short of a full rotation or go through an irregular shape when retracting the radius to avoid hardware / joint limits.

[0038] The orbital scanning mode 14 allows the stereoscopic camera 12 to be moved anywhere in 3D space (via the robotic arm 24) and allows for varying working distances while being locked and focused in real time on a specific point on the target site 16 (e.g., the target point 148 shown in FIG. 2). The orbital trajectory 230 may be defined in a spherical coordinate system with a first spherical angle (U) and a second spherical angle (V) as shown in FIG. 5 for an example of a location T in XYZ space and a projection Q onto the XY plane. FIG. 5 shows an exemplary virtual sphere 300 that can be employed by the controller C. The controller C and / or the robotic arm controller 42 of FIG. 1 allow the operator to move the stereoscopic camera 12 on the outer surface 302 of the virtual sphere 300 to an end position 306 while keeping the stereoscopic camera 12 pointed toward the center 304 (as shown by the view vectors 118, 118B). Center 304 has the same coordinates in XYZ space as a selected point on target site 16, such as target point 148 shown in FIG.

[0039] 6, a flow chart of an exemplary method 400 for implementing the orbital scanning mode 14 of FIG. 1 is shown. The method 400 may be embodied as computer readable code or instructions stored in and partially executable by the controller C of FIG. 1. The method 400 does not have to be applied in the particular order described herein, but may be executed dynamically. Furthermore, it should be understood that some steps may be omitted.

[0040] The method 400 starts at block 402 of FIG. 5, where the stereoscopic camera 12, via the robot arm 24 of FIG. 1, is centered on the reference plane 240 (see FIG. 3) of the eye 200. In some embodiments, referring to FIG. 3, the reference plane 240 extends through the fovea 242. In other embodiments, the reference plane 240 may extend through the optic disc 244. Also, according to block 402 of FIG. 6, the controller C is programmed to estimate a first working distance 246 (W) to the surface 248 of the eye 200, as shown in FIG. 3. The controller C is programmed to increase the radius or specific focal length F (see FIG. 2) to a second working distance 250 (see FIG. 3). The second working distance 250 may be obtained as the sum of the first working distance 246 and a first dimension R1, which may be a dimension of the eye, for example, a radius (anteroposterior direction) of the eye 200 obtained from an eye model and / or anatomical data of the patient of interest.

[0041] Proceeding from block 402 to block 404 of Fig. 6, the method 400 includes receiving input data such as joint position coordinates of the robot arm 24. At each update cycle, the controller C gets as input the current joint angle of the orbital trajectory 230, which may be different from the current joint angle of the robot arm 24. Also, per block 404, the controller C is adapted to transform the joint position coordinates from the camera coordinate system to a global coordinate system and to spherical coordinates defined by a first spherical angle (U) and a second spherical angle (V).

[0042] Proceeding from block 404 to block 406 of FIG. 6, the method 400 includes iterating through at least one of a first spherical angle (U) and a second spherical angle (V). First, the viewing angle of the orbital trajectory 230 may be changed by keeping the first spherical angle (U) fixed (e.g., at 0 degrees, 180 degrees, or any other angle) and iterating the second spherical angle (V) from (V+ΔV) to (V-ΔV) until a desired viewing angle 260 is reached. The desired viewing angle 260 represents the angle required to view the ora serrata 214 and may be preprogrammed into the controller C based on an eye model and / or anatomical data. In some embodiments, the change in viewing angle may be part of a "hold-to-move" input knob / device that may be adjusted to allow the surgeon to move on the trajectory at different viewing angles. The position of the target point 148 at the desired viewing angle 260 (V0) is locked via the controller C.

[0043] Second, the movement on the orbital trajectory 230 may be performed by repeatedly moving along a first sphere angle (U) of the virtual sphere 300 while keeping a second sphere angle (V) constant at the desired viewing angle 260. The start and end of the movement on the trajectory may be determined by a predefined starting angle (U initial ) and a predefined end angle (U final ) may be defined by the distance . Referring to FIG. 1, the robotic imaging system 10 may include an optical coherence tomography (OCT) module configured to acquire visualization data of the target site 16. The visualization data may be employed to guide the orbital trajectory 230, for example, via selection of a desired viewing angle 260. U.S. Patent Application No. 17 / 108,458, filed December 1, 2020, the contents of which are incorporated herein by reference in their entirety, describes a system having an integrated visualization camera and optical coherence tomography module.

[0044] Proceeding from block 406 to block 408 in FIG. 6, the controller C is configured to scale the velocity of the orbital trajectory 230 and calculate the end sphere point (e.g., end position 306 in FIG. 5). As described above, the orbital scanning mode 14 restricts the position of the stereoscopic camera 12 to the outer surface of a virtual sphere, for example the outer surface 302 of the virtual sphere 300 shown in FIG. 5, based on the current position of the target site 16. Iterative movement along the first spherical angle (U) allows the determination of the end position 306 for the trajectory input. The predetermined point on the outer surface 302 is given by an equation that is a function of the first spherical angle (U) and the second spherical angle (V). In other embodiments, the virtual sphere 300 may be represented by a different shape and the outer surface 302 may be represented by a flat surface 310. Based on the type of input received, the end position 306 (of the stereoscopic camera 12) on the outer surface 302 may be obtained in a different manner. For example, the input may be received via an input device 66 (e.g., a joystick or mouse) as shown in Fig. 1. The controller C and / or the robot arm controller 42 may be adapted to convert "up", "down", "left" and "right" from camera coordinates to robot base coordinate system coordinates, which are provided as X and Y vectors. The X and Y vectors are used by the controller C and / or the robot arm controller 42 to directly determine how the stereoscopic camera 12 moves on the virtual sphere 300 to determine the end position 306.

[0045] Also, per block 408, the controller C is adapted to calculate the amount of rotation required for the stereoscopic camera 12 to maintain lock on the coordinates of the target point 148 after the stereoscopic camera 12 has been moved. In other words, the controller C is configured to determine how the stereoscopic camera 12 should be oriented, given its new position on the virtual sphere 300, such that the view vector 118B of the end position 306 is provided at the same XYZ coordinates at the center of the virtual sphere 300 (corresponding to the selected point). The controller C and / or the robot arm controller 42 are adapted to determine the joint angles of the robot arm 24 and / or the coupling plate 26 required to achieve the desired orientation.

[0046] Proceeding from block 408 to block 410 of FIG. 6, the controller C is configured to apply some corrections and update the coordinates of the target point 148. The position of the target point 148 with coordinates [Xtarget, Ytarget, Ztarget] in the orbital trajectory 230 can be determined from the first spherical angle (U) of the eye 200 and the second dimension R2 as follows: [Xtarget=R2*Cosine(U), Ytarget=R2*Sine(U), Ztarget=fixed]. With reference to FIG. 3, the second dimension R2 of the eye 200 can be determined as a function of the desired viewing angle 260 (V0) and the dimension of the eye 200, e.g., the first dimension R1 (radius of the eye from anterior to posterior) as follows: [R2=R1*Sine(V0)]. With reference to FIG. 3, the second dimension R2 of the eye 200 may be based on a third dimension R3, which is the radius of the eye in the superior-inferior direction. The controller C may be configured to update the coordinate position [Xtarget, Ytarget, Ztarget] of the target point in the orbital trajectory 230 by, for example, injecting the respective difference values ​​at each update cycle. The corrections include anti-yaw corrections as well as roll and pitch amounts of the stereoscopic camera 12.

[0047] The correction (block 410 in FIG. 6) further includes an autofocus correction. U.S. Provisional Patent Application No. 63 / 167,406, filed March 29, 2021, describes an autofocus mode for a stereoscopic imaging system, the entire contents of which are incorporated herein by reference. The controller C may be programmed with one or more autofocus routines, such as a first autofocus routine 50 and a second autofocus routine 52 shown in FIG. 1. In the first autofocus routine 50, the orbital scanning mode 14 is adapted to maintain focus in the image while the robot arm 24 moves the stereoscopic camera 12 and the target site 16 is fixed. In the second autofocus mode 52, the orbital scanning mode 14 is adapted to maintain the image in focus as both the robot arm 24 and the target site 16 move. The position of the target point 148 [Xtarget, Ytarget, Ztarget] is continuously adjusted, for example, by automatically adjusting the focus motor to accommodate any total focal length change.

[0048] An exemplary implementation of the first auto focus routine 50 is described below. First, the controller C is programmed to calculate the change in working distance (ΔW) with respect to a previously stored value (e.g., a previous iteration value or an initialization value) due to the movement of the stereoscopic camera 12 via the robot arm 24, and obtain an updated value of the working distance (W). In other words, the displacement of the working distance (W) caused by the robot movement is tracked for each movement cycle. Second, the controller C is programmed to calculate and send a motor command for the focus motor 110 corresponding to the updated value of the working distance (W). A calibrated look-up table can be used to convert the change in working distance into a command for the focus motor 110. The radius of the virtual sphere 300 in each iteration is reset to the updated value of the working distance (W), with the internal calculation updating the radius of the virtual sphere 300 for each cycle. If the target site 16 is moving, the selected point (at the center 304 of the virtual sphere 300) is replaced with a dynamic trajectory corresponding to the movement of the target site 16. When all movements of the robot arm 24 are completed, the focus motor 110 is moved to the location of maximum sharpness. The amount of this adjustment (movement to the location of maximum sharpness) may be small due to the continuous tracking of the robot arm 24 during its movement. This results in an image that always appears in focus, even if the robot arm 24 is changing the working distance. The allowable change in the working distance W may be limited so that the radius of the virtual sphere 300 does not exceed the maximum and minimum working distances allowed by the hardware. In some embodiments, a feature flag may be inserted to enforce an artificial robot boundary to prevent the robot arm 24 from exceeding these limits on the working distance W.

[0049] An exemplary implementation of the second autofocus routine 52 is described below. The second autofocus routine 52 of FIG. 1 is adapted to keep the image in focus when the robot arm 24 moves, regardless of the type of movement, and to allow smooth low-noise maximum sharpness adjustment at the end of the movement. This means that the image remains in focus even if the stereoscopic camera 12 rotates to view a different scene, the scene itself changes depth, the working distance changes, or all of the above occur simultaneously. Even if the robot arm 24 moves and the target location, depth, and working distance may change, the second autofocus routine 52 measures the relative change in focal length and keeps the image in focus based on the deviation from the starting value. In other words, the second autofocus mode 52 requires an initial set of starting values ​​or estimates of the target position (in 3D space) and focal length. The initial set of starting values ​​can be provided to the controller C as the output of a subroutine or machine learning algorithm. For example, the starting values ​​can be obtained when the sharpness control routine is successfully executed once in application.

[0050] First, the controller C is programmed to determine the change in height due to the movement of the robot arm 24 (input in block 402). Height is defined as the change in position of the stereoscopic camera 12 along the axial direction (here the Z-axis) due to the movement of the robot arm 24. The controller C can calculate the change in height using position data of the joints (e.g., the joint sensors 33 in FIG. 1) and other parts of the robot arm 24.

[0051] Second, the controller C is programmed to determine a change in the target depth (ΔZ_target). The controller C may receive input data related to the disparity signal to calculate the change in the target depth. The change in the target depth (ΔZ_target) may be calculated using a feedback control with a closed loop control module, which may be a PI controller, a PD controller, and / or a PID controller. In one example, the change in the target depth (ΔZ_target) is calculated using a PID controller and a disparity value as follows: ΔZ_target=Kp(Rc-Rt)+Ki∫(Rc-Rt)dt-Kd×dRc / dt where Rc is the current disparity value and Rt is the initial target disparity value defined as the disparity value recorded when the starting value was initialized and stored. Here, Kp, Ki, and Kd are the proportional, integral, and derivative constants, respectively, from the PID controller, and the process variable is the difference between the current disparity value (Rc) and the initial target disparity (Rt). The constants Kp, Ki, and Kd can be obtained via calibration with a known change in target depth (here, along the Z-axis).

[0052] Third, the controller C is programmed to determine the change in position coordinates of the target site 16 based on the change in target depth. The stored position of the Z component of the target site 16 is updated as follows: Z_updated=Z_initial-ΔZ_target

[0053] Finally, the controller C is programmed to determine the composite focal length change (ΔF) and update the specific focal length (F). At each update cycle, the specific focal length F is updated using a feedforward term from the robot arm 24 plus a feedback term from the target depth disparity as follows: ΔF=ΔF_robot+ΔF_target The specific focal length F is updated depending on two factors: first, how much the end point of the view vector 118 of the stereoscopic camera 12 has changed, and second, how much the target depth of the target 16 has changed. Since the two factors are in different coordinate systems, a homogeneous transformation matrix (4×4) is used to transform the base coordinates (Xbase, Ybase, Zbase) in the robot base coordinate system to the camera coordinate system. The base coordinates (Xbase, Ybase, Zbase) represent the instant or current position of the target part 16 in the robot base coordinate system. The homogeneous transformation matrix is ​​composed of a rotation matrix Q (3×3) and a translation vector P, which can be expressed as follows:

number

number

number

number

[0054] Proceeding from block 410 of FIG. 6 to block 412, the controller C is configured to send updated motor commands for executing the orbital trajectory 230. The motor commands corresponding to the updated value of the particular focal length F are calculated and sent. The focus motor 110 is moved the correct amount, determined by calibration, so that the working distance is the same as the updated value of the particular focal length F. As described above, the orbital trajectory 230 is started at a predefined starting angle (U initial ) and a predefined end angle (U final ) while keeping the second spherical angle (V) constant at the desired viewing angle 260. When the robot arm 24 stops moving at the end of the orbital trajectory 230, the controller C is configured to determine a motor command for the focus motor 110 corresponding to a maximum sharpness position. The maximum sharpness position is based on one or more sharpness parameters including the sharpness signal, the maximum sharpness signal, and a time derivative of the maximum sharpness. In some embodiments, the sharpness signal is defined as the contrast between respective edges of objects in the stereoscopic image. The maximum sharpness signal is defined as the maximum sharpness value observed during the scanning period.

[0055] The controller C of Figure 1 may include or otherwise access information downloaded from remote sources and / or executable programs. With reference to Figure 1, the controller C may be configured to communicate with a remote server 60 and / or a cloud unit 62 via a network 64. The remote server 60 may be a private or public information source maintained by an organization such as, for example, a research institute, a company, a university, and / or a hospital. The cloud unit 62 may include one or more servers hosted on the Internet for storing, managing, and processing data.

[0056] The network 64 may be a serial communication bus in the form of a local area network. The local area network may include, but is not limited to, a Controller Area Network (CAN), Controller Area Network with Flexible Data Rate (CAN-FD), Ethernet, blue tooth, WIFI, and other forms of data. The network 64 may be a wireless local area network (LAN) that links multiple devices in a wirelessly distributed manner, a wireless metropolitan area network (MAN) that connects multiple wireless LANs, or a wireless wide area network (WAN) that covers a large area such as a nearby city or town. Other types of connections may also be used.

[0057] The controller C of FIG. 1 may be an integral part of the robotic imaging system 10 or may be a separate module operably connected to the robotic imaging system 10. The controller C comprises a computer-readable medium (also referred to as a processor-readable medium), including non-transitory (e.g., tangible) media involved in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memories. Volatile media may include, for example, dynamic random access memory (DRAM), which may constitute a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wires, and optical fibers, including the wires that comprise a system bus coupled to the processor of the computer. Some forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, other magnetic media, CD-ROM, DVD, other optical media, punch cards, paper tape, other physical media with patterns of holes, RAM, PROM, EPROM, Flash EEPROM, other memory chips or cartridges, or other media from which a computer can read.

[0058] The lookup tables, databases, data repositories, or other data stores described herein may include various types of mechanisms for storing, accessing, and retrieving various types of data, including a hierarchical database, a set of files in a file system, a proprietary application database, a relational database management system (RDBMS), and the like. Each such data store may be contained within a computing device employing a computer operating system, such as one of the computer operating systems described above, and may be accessed over a network in one or more of a variety of ways. The file system may be accessible from the computer operating system and may include files stored in various formats. The RDBMS may use a structured query language (SQL) in addition to a language for creating, saving, editing, and executing stored procedures, such as the PL / SQL language described above.

[0059] The flowcharts presented herein illustrate the architecture, functionality, and operation of anticipated implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of code, and includes one or more executable instructions for implementing the specified logical functions. It should also be noted that each block of the block diagrams and / or flowchart examples, and combinations of blocks in the block diagrams and / or flowchart examples, may be implemented by a special purpose hardware-based device that performs the specified functions or acts, or a combination of special purpose hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium. The computer program instructions, including instructions for implementing the functions / acts specified in the flowchart and / or block diagram blocks, may instruct a controller or other programmable data processing apparatus to function in a specific manner to produce a product, as stored in the computer-readable medium.

[0060] Numerical values ​​of parameters (e.g., amounts or conditions) in this specification, including the appended claims, are to be understood as being modified by "about" in each and every instance, regardless of whether the term "about" is actually placed before the numerical value. "About" indicates that the numerical value described allows for some slight imprecision (close to the exact value, roughly or reasonably close, approximately). If the imprecision provided by "about" is not understood in this ordinary sense in the art, "about" as used herein at least indicates the variation that may result from the ordinary method of measuring and using such parameters. In addition, the disclosure of a range includes the disclosure of each value and each sub-range within the entire range. Each value within the range and the endpoints of the range are disclosed herein as separate embodiments.

[0061] The detailed description and drawings or figures support and explain the disclosure, but the scope of the disclosure is defined only by the claims. Although some of the best modes and other embodiments for carrying out the claimed disclosure have been described in detail, there are various alternative designs and embodiments for carrying out the disclosure defined in the appended claims. Furthermore, the features of the embodiments shown in the drawings or the various embodiments mentioned herein should not necessarily be understood as independent embodiments from each other. Rather, each of the characteristics described in one of the example embodiments can be combined with one or more other desirable characteristics from the other embodiments, resulting in other embodiments not described in words or by reference to the drawings. Therefore, such other embodiments are included within the framework of the appended claims.

Claims

1. 1. A robotic imaging system for imaging a target site in an eye, the robotic imaging system comprising: a stereoscopic camera configured to record left and right images of the target area to generate at least one stereoscopic image of the target area; a robotic arm operatively connected to the stereoscopic camera, the robotic arm adapted to selectively move the stereoscopic camera relative to the target site; a robotic arm, the stereoscopic camera comprising an optical assembly having at least one lens and defining a working distance, the optical assembly having at least one focus motor adapted to move the at least one lens to selectively vary the working distance; a controller in communication with the robotic arm and having a processor and a tangible, non-transitory memory having instructions stored thereon, the controller configured to determine a change in target depth from an initial target position, the change in target depth being defined as a displacement of a position of the target site along an axial direction; the controller is configured to update a particular focal length based in part on the change in the target depth; a controller adapted to selectively execute an orbital scanning mode to cause the robotic arm to sweep an orbital trajectory at least partially circumferentially around the eye while maintaining focus; A robot imaging system comprising:

2. The robotic imaging system of claim 1 , wherein the target site comprises the ora serrata of the eye.

3. the orbital orbit is defined in spherical coordinate axes to define a first spherical angle and a second spherical angle; 2. The robotic imaging system of claim 1, wherein the controller is adapted to vary the viewing angle of the orbital track by holding the first spherical angle constant while iterating the second spherical angle until a desired viewing angle is reached.

4. 4. The robotic imaging system of claim 3, wherein the controller is adapted to selectively direct the orbital trajectory by cycling the first spherical angle between a predefined start angle and a predefined end angle while holding the second spherical angle constant at the desired field of view angle.

5. The robotic imaging system of claim 1 , wherein the orbital track at least partially forms a circle.

6. The robotic imaging system of claim 1 , wherein the orbital trajectory at least partially forms an ellipse.

7. The robotic imaging system of claim 1 , wherein the orbital trajectory ranges from about 180 degrees to 300 degrees.

8. The robotic imaging system of claim 1 , wherein the orbital trajectory subtends an angle of approximately 360 degrees.

9. the controller is configured to center the stereoscopic camera about the ocular reference surface and estimate a first working distance relative to the ocular reference surface; The robotic imaging system of claim 1 , wherein the controller is adapted to change a view vector of the stereoscopic camera to a desired viewing angle.

10. 2. The robotic imaging system of claim 1, wherein the controller is configured to lock the respective positions of each target point along the orbital trajectory by constraining the respective positions of the stereoscopic cameras to an outer surface of a virtual sphere, the virtual sphere defining a radius equal to the particular focal length.

11. The robotic imaging system of claim 1 , wherein the particular focal length is based in part on a desired angle of view, a size of the eye, and a first working distance.

12. the controller is configured to determine a change in height of the stereoscopic camera from an initial camera position, the change in height being defined as a displacement of a position of the stereoscopic camera along an axial direction; The robotic imaging system of claim 10 , wherein the controller is configured to update the particular focal length based in part on the change in the height of the stereoscopic camera.

13. when the robot arm is no longer moving, the controller is configured to determine a motor command for the at least one focus motor that corresponds to a maximum sharpness position; The robotic imaging system of claim 1 , wherein the position of maximum sharpness is based on one or more sharpness parameters including a sharpness signal, a maximum sharpness signal, and a time derivative of maximum sharpness.

14. The robotic imaging system of claim 1 , wherein in each update cycle, the controller is configured to inject a respective difference value into each coordinate location of the orbital trajectory.

15. 1. A stereoscopic imaging system for imaging a target site in an eye, the stereoscopic imaging system comprising: a stereoscopic camera configured to record left and right images of the target area to generate at least one stereoscopic image of the target area; a robotic arm operatively connected to the stereoscopic camera, the robotic arm adapted to selectively move the stereoscopic camera relative to the target site; a robotic arm, the stereoscopic camera comprising an optical assembly having at least one lens and defining a working distance, the optical assembly having at least one focus motor adapted to move the at least one lens to selectively vary the working distance; a controller in communication with the robotic arm and having a processor and a tangible, non-transitory memory having instructions stored thereon, a controller adapted to selectively execute an orbital scanning mode to cause the robotic arm to sweep an orbital trajectory at least partially circumferentially around the eye while maintaining focus; A stereoscopic imaging system comprising:

16. The stereoscopic imaging system of claim 15 , wherein the target site comprises the ora serrata of the eye.

17. the orbital orbit is defined in spherical coordinate axes to define a first spherical angle and a second spherical angle; the controller is adapted to vary a field of view of the orbital track by holding the first spherical angle constant while iterating the second spherical angle until a desired field of view is reached; 16. The stereoscopic imaging system of claim 15, wherein the controller is adapted to selectively direct the orbital trajectory by cycling the first spherical angle between a predefined start angle and a predefined end angle while holding the second spherical angle constant at the desired field of view angle.

18. when the robot arm is no longer moving, the controller is configured to determine a motor command for the at least one focus motor that corresponds to a maximum sharpness position; 16. The stereoscopic imaging system of claim 15, wherein the position of maximum sharpness is based on one or more sharpness parameters including a sharpness signal, a maximum sharpness signal, and a time derivative of maximum sharpness.

19. the sharpness signal is defined as the contrast between respective edges of objects in the at least one stereoscopic image; and 20. The stereoscopic imaging system of claim 18, wherein the maximum sharpness signal is defined as the maximum sharpness value observed during a scan period.

Images