Robotic 3D display mount for digital imaging of ophthalmic surgery

Abstract

The system includes a display device and an actuation structure mounted on the display device and configured to move the display device in three-dimensional space. The system includes an imaging device configured to capture an image of the patient's eye during ophthalmic surgery and display the image on the display device. A controller is configured to actuate the actuation structure to maintain visibility of the display device to the surgeon performing the ophthalmic surgery. The display may be positioned based on the surgeon's profile and / or treatment plan. Visibility of the display may be determined based on images from the imaging device, the display device, or a camera mounted elsewhere. Visibility of the display may also be determined based on kinematic data of the actuation structure and / or supports for the imaging device.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 505,050, filed May 30, 2023, which is hereby assigned to the assignee hereof and is hereby expressly incorporated by reference in its entirety into this document as if fully set forth below.

Background Art

[0002] Introduction Ophthalmic surgery is performed to treat cataracts, glaucoma, and retinal diseases. During ophthalmic surgery, surgeons generally peer through the eyepiece of a binocular surgical microscope to view the intricate anatomical structures of the patient's eye and surgical instruments. However, bending over the surgical microscope for hours causes neck injuries to many surgeons.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Improving the ergonomics of ophthalmic surgery to prevent such injuries would be an advancement in the art.

Means for Solving the Problems

[0004] In some embodiments, a system is provided. The system includes a display device and an actuating structure mounted to the display device and configured to move the display device in a three - dimensional space. The system includes an imaging device configured to capture an image of a patient's eye during ophthalmic surgery and display the image on the display device. A controller is configured to operate the actuating structure to maintain visibility of the display device for a surgeon performing ophthalmic surgery.

[0005] In some embodiments, a system is provided. The system includes a display device and an actuation structure mounted on the display device and configured to move the display device in three-dimensional space. The system further includes an imaging device configured to capture surgical images of a patient's eye during ophthalmic surgery and display the surgical images on the display device, and a camera mounted on the imaging device. A controller is configured to receive camera images from the camera, evaluate the representation of the display device in the camera images, and, based on the representation of the display device, activate the actuation structure to keep the display device unobstructed within the field of view of the surgeon performing the ophthalmic surgery.

[0006] To allow for a more detailed understanding of the features of this disclosure, a more detailed description of this disclosure, as summarized above, may be made by reference to embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings show only exemplary embodiments and are not intended to limit the scope, and other equally effective embodiments may be recognized. [Brief explanation of the drawing]

[0007] [Figure 1] This section describes exemplary operating environments, including robotic three-dimensional display mounts according to several embodiments. [Figure 2] Figure 1 is a side view of an exemplary operating environment, illustrating alternative mounting options for a robotic three-dimensional display mount according to several embodiments. [Figure 3] Figure 1 is a top view of an exemplary operating environment, showing various positions of a three-dimensional display according to several embodiments. [Figure 4] This is a schematic block diagram of inputs for controlling a robotic three-dimensional display mount according to several embodiments. [Figure 5] This is a process flow diagram of a method for controlling a robotic three-dimensional display mount according to several embodiments. [Figure 6] This figure shows image processing for controlling the positioning of a robotic three-dimensional display mount according to several embodiments. [Figure 7] This section describes an exemplary computing device that at least partially performs one or more functions for controlling the positioning of a robotic three-dimensional display mount according to several embodiments. [Modes for carrying out the invention]

[0008] For the sake of clarity, the same reference numerals are used to refer to identical elements common to the drawings where possible. Elements and features of one embodiment are considered to be conveniently incorporated into other embodiments without further description.

[0009] Figure 1 shows an exemplary system 100 in which a robotic display mount may be used. System 100 includes an ophthalmic microscope 102. A surgeon 104 uses the ophthalmic microscope 102 to visualize the structure on and inside the eye 106 of a medical patient 108 undergoing surgery or examination. In this figure, the microscope 102 is supported by an adjustable overhead arm 110 of a microscope support stand 112. The patient 108 may be supported by an operating table 114. Since the ophthalmic microscope 102 is movable with the overhead arm 110 in three translational dimensions and possibly one or more rotational directions, the surgeon 104 can position the ophthalmic microscope 102 as desired relative to the eye 106 of the patient 108.

[0010] In some embodiments, the ophthalmic microscope 102 includes a high-resolution, high-contrast stereoscopic surgical microscope. The ophthalmic microscope 102 often includes a monocular eyepiece 116 or a binocular eyepiece 116 through which the surgeon 104 obtains an optically magnified view of the relevant ocular structures that the surgeon 104 needs to see in order to perform a given surgery or diagnosis of the eye condition of the patient 108.

[0011] The ophthalmic microscope 102 includes a digital camera and broadband light source for capturing color (red, green, and blue) images, a multispectral imaging (MSI) device, and / or other types of imaging devices. The digital images captured using the camera may be displayed on a display device within the ophthalmic microscope 102.

[0012] The ophthalmic microscope 102 may include two display devices that can be viewed through binocular eyepieces 116, which display images of the patient's eye 106 captured from different viewpoints by two cameras to provide stereoscopic vision. For example, the ophthalmic microscope 102 may be implemented as the NGENUITY 3D VISUALIZATION SYSTEM provided by Alcon Inc. (Fort Worth, Texas).

[0013] In addition to or instead of the above, images from the ophthalmic microscope 102 may be displayed on one or more display devices. For example, one or more display devices may include a display device 118 fastened to a support arm 110 above the ophthalmic microscope 102.

[0014] To free the surgeon from the need to constantly look through the eyepiece 116 to obtain stereoscopic vision, the display device 120 can be realized as a three-dimensional display device. Therefore, the display device 120 can provide stereoscopic vision of images captured using the ophthalmic microscope 102. The display device 120 can be embodied as any type of three-dimensional display device known in the art, including those that use special filtering glasses or those that do not. With respect to some types of three-dimensional display devices, three-dimensional perception requires that the observer's distance from the display device 120 be within a threshold distance from the display device.

[0015] The display device 120 may, conveniently, be large enough to allow observation of the anatomical structure of the patient's eye 106 and important details of surgical instruments. For example, the size of the display device 120 may be greater than 35 inches in diagonal length, 45 inches in diagonal length, 55 inches in diagonal length, 65 inches in diagonal length, or larger. The resolution of the display device 120 may be at least 1080p, 1440p, 2160p, 2540p, 4000p, or 4320p as defined by the Advanced Television Systems Committee (ATSC), or any other customary resolution. The display device 120 is preferably a color display device 120 and may be implemented as a light-emitting diode (LED) display, liquid crystal display (LCD), organic LED (OLED) display, display light projection (DLP), or any other type of display technology.

[0016] The display device 120 can help improve the surgeon's posture to help avoid neck strain and injury, and positioning the display for each procedure takes several minutes. Similarly, the movement of the surgeon 104, the ophthalmic microscope 102, or other personnel and equipment may necessitate repositioning of the display device 120 during surgery. In some cases, if the display device 120 becomes obstructed, the surgeon 104 may revert to using the eyepiece 116 of the ophthalmic microscope 102, thereby negating the benefits of the display device 120.

[0017] The display device 120 is mounted on a robotic arm 122, which is controlled to keep the display device 120 visible to the surgeon at the start of surgery and to adjust the position of the display device 120 during surgery to maintain visibility of the display device 120. As used herein, “maintain visibility” may be understood to include keeping the line of sight to at least the entire display device, and also to maintain the distance and orientation of the display device 120 relative to the surgeon’s eye to either or both of the following: (a) within the surgeon’s preferred tolerance for distance and orientation, or within other predetermined tolerances for distance and orientation, and (b) enabling the surgeon to perceive the three-dimensional image displayed on the display device 120 as may be specified by the manufacturer of the display device 120.

[0018] The robotic arm 122 may have 4, 5, or 6 degrees of freedom (DOF). For example, the robotic arm 122 may be embodied as a 6DOF serial robotic arm. The robotic arm 122 may be mounted on a base 124 which is mounted on the ceiling or elsewhere in the operating room, including the system 100. The robotic arm 122 may also be mounted on a mobile cart which can be positioned adjacent to the operating table 114. A first link 126 is attached to the base 124 by a shoulder joint 128. For example, the joint 128 may be a rotary joint that allows the joint 128 to rotate from 45 to 360 degrees. A rotary coupling may be used to couple signals to the actuators and display device 120 of the robotic arm 122, allowing rotations greater than 360 degrees to be performed.

[0019] Link 126 can be connected to link 132 by joint 130. Link 132 can be connected to link 136 by joint 134, and link 136 can be connected to link 140 by joint 138. Joints 130, 134, and 138 are embodied as elbow joints in the illustrated embodiment. Link 140 can be coupled by joint 142 to a mounting point to the display device 120. Joint 142 can be a rotational joint that allows the display device 120 to rotate about an axis with respect to link 140.

[0020] In the illustrated example, five joints 128, 130, 134, 138, and 142 are shown, thereby enabling 5DOF, which is sufficient to allow the display device 120 to be positioned considering typical ophthalmic procedures of the surgeon 104. If the robotic arm 122 is a commercially available robotic arm, 6DOF can be made available with the sixth DOF not being used. Alternatively, all 6DOF may be used to provide more flexibility in determining the kinematic solution for positioning the display device 120 in a desired position and orientation, for example, to reduce the installation area, expand the range of reachable positions, or otherwise improve the performance of the robotic arm 122. For example, the sixth DOF can allow the display device 120 to rotate about a line perpendicular to the screen of the display device 120, which is not necessary in most applications. However, in other embodiments, such rotation is used, for example, to change the orientation of the larger dimension of the display between a vertical and a horizontal orientation. In other embodiments, the sixth DOF can be used to change the tilt of the display device 120 in a plane parallel to the Z direction (see the definition of the Z direction in FIGS. 2 and 3).

[0021] The illustrated robotic arm 122 is an example of an actuating structure that can be used to control the position and orientation of the display device 120. Other actuating structures, such as other types of robotic arms, gantries, or other assemblies of actuators and links, including those that include translational joints instead of or in addition to rotational joints, may also be used.

[0022] Referring to FIG. 2, the robotic arm 122 may be mounted to the overhead arm 110 at the illustrated position 200 or to the floor at the illustrated position 202. The robotic arm 122 may also be mounted to a mobile cart that can be rolled against a wall, the table 112, the operating table 114, or the floor.

[0023] As disclosed in more detail below, the robotic arm 122 may be automatically controlled to maintain the screen of the display device 120 visible to both eyes 206 of the surgeon 104. Therefore, the robotic arm 122 may be actuated to move the display device 120 to a position facing the current position of the surgeon without being blocked by the ophthalmic microscope 102 or other obstacles.

[0024] The positions of the surgeon's eyes 206 and the display device 120 may be defined with respect to the X (horizontal), Y (longitudinal), and Z (vertical, i.e., the direction of gravity) directions that are mutually orthogonal. The operating table 114 may be adjusted in the Z direction taking into account the height of the patient's head 208 so that the patient's eyes 106 are at the desired height for the surgeon 104 to perform surgery. The position of the display device 120 in the Z direction may be automatically controlled so that the ophthalmic microscope 102 is not blocked by the line of sight (ray) extending between the surgeon's eyes 206 and any point on the screen of the display device 120. The position of the display device 120 in the X-Y plane may be selected to avoid other potential obstacles, such as the link 210 suspending the ophthalmic microscope from the overhead arm 110, other personnel present in the operating room, other equipment, or the like.

[0025] Automatic positioning of the robotic arm 122 can be facilitated by one or more cameras. In one embodiment, one or more cameras 212 may be mounted on a medical microscope 102, with their field of view facing outward from the surgeon 104. The screen of the display device 120 is within the field of view of the camera 212, and the output of the camera 212 is used to estimate the relative position of the ophthalmic microscope 102 and the display device 120. Thus, the relative position can be used to adjust the position of the display device 120 using the robotic arm 122 so that the display device 120 is positioned over a larger area within the field of view of the surgeon's eye.

[0026] In other embodiments, the camera 214 is mounted on the display device 120, for example, around the screen of the display device 120. The output of the camera 214 may be evaluated to determine whether the surgeon's eye 206 is within the field of view of each camera 214. The position of the display device 120 may be adjusted using a robotic arm 122 to ensure that the surgeon's eye 206 is within the field of view of all cameras 214.

[0027] In yet another embodiment, one or more cameras 216 may be mounted in one or more locations within the operating room, for example, on an overhead arm 110, a table 112, one or more walls, or the ceiling. The output of the cameras 216 may be used to determine the position of the surgeon's eye 206, the orientation of the surgeon's head, the position and orientation of the ophthalmic microscope 102, the position of other potential obstructions, and / or the position and orientation of the display device 120. Thus, the position and / or orientation information obtained using the cameras 216 may be used, if necessary, to adjust the position and orientation of the display so that the display device 120 is positioned over a larger area within the field of view of the surgeon's eye.

[0028] Referring to Figure 3, the display device 120 can be positioned in various locations depending on the procedure performed by the surgeon 104. For example, when performing retinal surgery, the surgeon 104 may sit in the upper position shown in the figure. When performing cataract surgery, the surgeon 104 may sit in the left temporal position 104a, with the ophthalmic microscope 102 in position 102a, and perform surgery on the left eye; or sit in the right temporal position 104b, with the ophthalmic microscope 102 in position 102b, and perform surgery on the right eye.

[0029] The display device 120 is automatically positioned by the robotic arm 122 based on the procedure being performed, avoiding any delays required for manual positioning and repositioning, without the need for the surgeon or other operating surgeon to do so. For example, the display device 120 may be in the illustrated position when the surgeon 104 is in a superior position. The display may be in position 120a when the surgeon is in a left temporal position, and in position 120b when the surgeon is in a right temporal position. As is evident, in the illustrated position, the screen of the display device 120 is substantially parallel (e.g., within 5 degrees) to the XZ plane. In positions 120a and 120b, the screen of the display is substantially parallel to the YZ plane. During use, the display device 120 may be moved by the robotic arm 122 within a range of positions around the indicated object to avoid obstacles.

[0030] Figure 4 shows components and data that may be used to control the robotic arm 122 in order to maintain visibility of the display device 120 to the eye 206 of the surgeon 104. The control algorithm 400 may be executed by a controller which is embodied as a separate computing device or as a computing device incorporated into one or more of the ophthalmic microscope 102, the robotic arm 122, and the display device 120.

[0031] The control algorithm 400 may receive the kinematic state 402 of the robot arm 122 as input. As used herein, “kinematic state” can be understood as the relative orientation of the links of the robot arm, and possibly the velocity and acceleration of the links of the robot arm. The kinematic state may include, or be sufficient to derive, the position and orientation of the end of the robot arm in three-dimensional space. For example, the kinematic state 402 may include the relative positions and orientations of the links 126, 132, 136, and 140 of the robot arm 122 relative to each other, to the base 124, and to the display device 120. The kinematic state 402 may further include the velocity and acceleration of the links 126, 132, 136, and 140. The kinematic state 402 may include, or be sufficient to derive, the position and orientation of the display device 120 in three-dimensional space.

[0032] The control algorithm 400 may receive as input the kinematic state 404 of some or all of the supports of the ophthalmic microscope 102, such as the base 112, the overhead arm 110, the link 210 suspending the ophthalmic microscope from the overhead arm 110, the joints between these members, and any other links or joints supporting the ophthalmic microscope 102. The kinematic state 404 may include the relative position and orientation of the base 112, the overhead arm 110, the link 210, or one or more other links and joints. The kinematic state 404 may further include the velocity and acceleration of some or all of the base 112, the overhead arm 110, the link 210, or one or more other links and joints. The kinematic state 404 may include, or be sufficient to derive, the position and orientation of the ophthalmic microscope 102 in three-dimensional space.

[0033] In embodiments where kinematic states 402 and 404 are available and provide the position and orientation of the ophthalmic microscope 102 and the display device 120 in three-dimensional space, whether or not the display device 120 is obstructed can be determined solely from states 402 and 404. For example, whether or not the display device 120 is obstructed can be determined from inputs including the known position and orientation of the ophthalmic microscope 102 and the display device 120, the assumed position of the surgeon's eye 206 relative to the ophthalmic microscope 102, and possibly some or all of the other information from the treatment plan 414, such as the working distance of the microscope relative to the patient's eye 106, the height of the patient's bed, and the height of the patient's head. Using these inputs, it can be determined whether the surgeon's eye 206 is obstructed by the ophthalmic microscope 102, the link 210, or any other link supporting the ophthalmic microscope 102, and a new position for the display device 120 can be calculated (e.g., using kinematic reachability calculations) so that the robotic arm 122 is commanded to achieve the new position.

[0034] In embodiments where kinematic conditions 402, 404 are unavailable, or where a more accurate assessment of the visibility of the display device 120 is required or desired, the control algorithm 400 may further acquire, as input, images output from some or all of the following: one or more cameras 212 mounted on the ophthalmic microscope 102, one or more cameras 214 mounted on the display device 120, and one or more other cameras 216 mounted in the operating room, on the overhead arm 110, the table 112, or elsewhere. The control algorithm 400 receives images from some or all of the cameras 212, 214, 216, estimates the relative position of the display device 120 and the surgeon's eye 206, estimates any obstructions to the display, and determines appropriate adjustments to the position and orientation of the display device 120 to remove the obstructions. For example, at any point during surgery, the surgeon may be in a kyphosis, a reclined position, or some intermediate position, or any position in between, which affects the position (particularly height) and gaze direction of the surgeon's eye. Therefore, tracking the position and gaze direction of the surgeon's eye 206 using images from one or more cameras 216 can be used to maintain visibility of the display device 120 when the surgeon changes position.

[0035] The control algorithm 400 further takes data as input to facilitate the determination of the initial position of the display device 120. For example, the control algorithm 400 may receive either or both of the surgeon profile 412 and the treatment plan 414. The surgeon profile 412 corresponds to the surgeon 104 performing the procedure using the ophthalmic microscope 102 and the display device 120. The surgeon profile 412 may include the surgeon's height, the height of the surgeon's eye 206 when seated and performing the procedure, the preferred position of the ophthalmic microscope relative to the surgeon's eye 206, or other information. The surgeon profile 412 may include measurements of the position of the surgeon's eye 206 relative to the ophthalmic microscope or in a particular procedure. The surgeon profile 412 may include the preferred viewing angle of the surgeon's eye 206. The measurements of the surgeon profile 412 may be obtained manually or determined from images obtained using cameras 214, 216 during a previous procedure.

[0036] The treatment plan 414 specifies the steps of the procedure to be performed. The treatment plan 414 may include textual guidance for the surgeon, an overlay superimposed on images captured using an ophthalmic microscope 102, or other information. The treatment plan 414 may indicate the eye in which the surgery will be performed and the type of procedure, e.g., retinal surgery, cataract surgery, glaucoma surgery. The surgeon's position (superior, left temporal, right temporal) may be inferred from the treatment plan 414, or the treatment plan 414 may indicate the surgeon's position. In some cases, the treatment plan 414 may require the surgeon to change position, such as from superior to left temporal. This change may be explicitly defined or may be determined by the treatment specified in the treatment plan.

[0037] The control algorithm 400 receives the treatment plan 414 and, at the start of the surgery, determines the position of the display device 120 in accordance with the surgeon's position (see, for example, Figure 3 and the corresponding description). The control algorithm 400 may interface with an ophthalmic microscope, which presents guidance according to the treatment plan 414. If the treatment plan reaches a stage that requires the surgeon to change to a new position, the control algorithm 400 may detect the change and adjust the position and orientation of the display device 120 to correspond to the new surgeon's position.

[0038] The control algorithm 400 may further receive surgeon input 416. Surgeon input 416 may include buttons, touchscreens, voice control, gesture control, or other interfaces for receiving input from surgeon 104. The surgeon may use input 416 to directly command a change in the display device 120 to a position corresponding to the superior, left-temporal, or right-temporal position. The surgeon or other operator may also use input 416 to command the robotic arm 122 to adjust the position and orientation of the display device 120 (e.g., up and down, left, right, or rotation in the XY plane or in a plane parallel to the Z direction). The surgeon may use input 416 to suppress or trigger automatic adjustment of the position and orientation of the display device 120. In some embodiments, the robotic arm 122 includes force and / or torque sensors so that forces applied to the display 120 or the robotic arm 122 by the surgeon or other operator can be detected and used as inputs to trigger a change in the position and / or orientation of the display 120.

[0039] The control algorithm 400 produces an output that is transmitted to the actuators 418 of the joints of the robot arm 122, such as joints 128, 130, 134, 138, 142, or other joints in other types of robot arms, in the example of Figure 1. The output of the control algorithm 400 may also be in the form of a desired position and orientation of the display device 120, and a separate algorithm may determine to change the angles of the joints to achieve the position and orientation, for example, according to a kinematic model of the robot arm 122 and sensors that provide the current kinematic state 402 of the robot arm 122.

[0040] Referring to Figure 5, the control algorithm 400 may perform the illustrated method 500 on an ophthalmic microscope 102, a surgeon 104, a display device 120, and a robotic arm 122, and possibly one or more cameras 212, 214, 216.

[0041] Method 500 includes, in step 502, receiving a surgeon profile 412; in step 504, receiving a treatment plan 414; and in step 506, determining the initial position and orientation of the display device 120 relative to the ophthalmic microscope 102. For example, the surgeon profile 412 indicates the height of the surgeon's eye 206 and the preferred height of the ophthalmic microscope 102. The treatment plan 414 indicates the surgeon's position (upper, left temporal, right temporal). Thus, the initial position is determined as the position and orientation in the XY plane facing the surgeon's position (see Figure 3 and the corresponding description), as well as the height in the Z direction selected so that the screen of the display device 120 appears at a given height of the surgeon's eye, at the height of the microscope 102, and possibly based on the position of the link 210 or other known structure. The initial position may include selecting an inclination in a plane parallel to the Z direction and an angle in the XY plane so that the normal vector of the screen is oriented approximately (e.g., within 5 degrees) towards the planned position of the surgeon's eye. For example, the planned position of the surgeon's eye 206 may be at the height of the surgeon's eye in the Z direction, as shown in the surgeon profile 412, or it may be at a predetermined offset from the ophthalmic microscope 102 in a plane intersecting the center of the ophthalmic microscope 102, for example, a plane passing between the eyepieces 116. The predetermined point may be the surgeon's preferred position relative to the ophthalmic microscope 102, as shown in the surgeon profile 412. If the display device 120 is a three-dimensional display device, the initial position may be selected to be within a threshold distance and threshold angle relative to the surgeon's eye 206 with respect to the ideal orientation, so as to enable three-dimensional perception.

[0042] Method 500 includes, in step 508, activating one or more actuators of the robotic arm 122 to achieve the initial position and orientation of the display device 120 as determined in step 506. If the treatment plan 414 specifies a change in the surgeon's position to a new position during the procedure, it should be noted that steps 506 and 508 may be repeated for the new position when the point in the procedure in which the change of position is specified is reached.

[0043] In some embodiments, including the use of cameras 214, 216 that provide images enabling the determination of the relative positions of the surgeon 104 and the ophthalmic microscope 102, steps 502-508 may be omitted. For example, the surgeon may sit in a desired position and the ophthalmic microscope 102 may be positioned in a desired position. The control algorithm may determine the positions of the surgeon's eye 206 and the ophthalmic microscope 102, the position of the display device 120 not obstructed by the ophthalmic microscope 102, the link 210, or other structures, and the normal vector of the screen of the display device 120 facing the surgeon's eye 206, for example, may be directed within 5 degrees of the surgeon's eye 206.

[0044] During the procedure, movements of the surgeon 104, the ophthalmic microscope 102, and other equipment or personnel that obstruct the surgeon's view of the display device 120 can be detected and compensated for by the movement of the display device 120. For example, Method 500 may include, in step 510, receiving the kinematic states 402, 404 of one or both of the robotic arm 122 and the arm supporting the ophthalmic microscope 102. Method 500 may also include, in step 512, receiving images from some or all of the cameras 212, 214, and 216. Note that in some embodiments, only one of steps 510 and 512 may be performed.

[0045] Method 500 includes, in step 514, estimating the visibility of the display device 120 based on (a) kinematic states 402, 404 and (b) images from some or all of the cameras 212, 214, 216, one or both of them.

[0046] Using kinematic states 402 and 404, the visibility of the display device 120 can be estimated by using the known positions of the ophthalmic microscope 102 and the display device 120, as well as the estimated position of the surgeon's eye 206 (e.g., from the surgeon's profile), to determine whether the ophthalmic microscope 102 is within the line of sight of the surgeon's eye 206.

[0047] When using camera 216, determining visibility can be similar to determining the methods used for kinematic conditions 402, 404. Images from camera 216 can be evaluated to determine the location of the ophthalmic microscope 102, the display device 120, the surgeon's eye 206, and any obstructions. Using this information, it can be determined whether the entire display device 120 is within the field of view of the surgeon's eye and unobstructed, for example, by using a line of sight that traces between the surgeon's eye and a set of points dispersed across the display device 120.

[0048] Continuing with reference to Figure 5, and then to Figure 6, step 514 may include using images from some or all of the cameras 212, 214, and 216 to estimate the visibility of the display device 120. For example, an image 600 captured by camera 212 includes a representation 602 of the display device 120. The representation of the display device 120 may be recognized by identifying an image captured by an ophthalmic microscope 102 displayed on the display device 120.

[0049] The position of the representation 602 in the image can be compared to a predetermined region 604 that corresponds to the position where the display device 120 may appear to the surgeon's eye 206. The region 604 may be universal to all users, or it may take into account the height of the surgeon's eye 206 as shown in the surgeon profile 412, for example. If the representation 602 of the display device 120 is not fully positioned within the region, the display device 120 may be determined to be obstructed.

[0050] Image 600 may be further analyzed to determine whether any portion of the image received from the ophthalmic microscope 102 is obscured within Image 600. For example, Image 600 may include a representation 606 of an obstruction to the visibility of the display device 120, which may be identified in step 514.

[0051] Using images from camera 214, step 514 may include evaluating whether the surgeon's eye 206 is visible in each image from each camera 214. If the surgeon's eye is not visible in images from at least one camera 214, the display device 120 may determine that it is obstructed.

[0052] If, in step 516, the display device 120 is found to be obstructed, the method 500 may include determining an unobstructed position in step 518. For example, continuing the example in Figure 6, step 518 may include determining that position 608 is within area 604 and does not overlap with the obstruction represented by representation 606. For example, step 518 may include moving the display device 120 in the opposite direction to the portion of representation 602 that is obstructed or outside area 604. For example, in Figure 6, the display device 120 may be moved upward in response to representation 602 extending below area 604. The display device 120 may be moved left in response to the right side of representation 602 that is obscured.

[0053] Using the image from camera 214, step 518 may include moving the display device 120 in the opposite direction from camera 216 whose received image does not have the surgeon's eye 206 in its field of view. For example, if camera 216 at the left edge of the display device 120 does not have the surgeon's eye 206 in its field of view, the display device 120 may be moved to the right. If camera 216 at the bottom edge of the display device 120 does not have the surgeon's eye 206 in its field of view, the display device 120 may be moved upward.

[0054] Using the positions determined using kinematic states 402, 404 and / or images from camera 216, step 518 may include determining a new position of the display device 120 in which the estimated visibility of the display device 120 is improved compared to the current estimated visibility, based on the known position of the ophthalmic microscope 102 and the estimated or detected position of the surgeon's eye 206. In some embodiments, both kinematic states 402, 404 and images from camera 216 are used. For example, images from camera 216 may be used to determine the position of obstacles other than the ophthalmic microscope 102. Therefore, step 518 may include determining the position and orientation of the display device 120 in which the display device is visible, based on the known position of the ophthalmic microscope 102 and any identified obstacles.

[0055] In any of the exemplary implementations described above in step 518, the determined position and orientation of the display device 120 may position the display device 120 within a threshold distance from the estimated or detected position of the surgeon's eye 206, thereby enabling stereoscopic vision, where the display device 120 is a three-dimensional display device.

[0056] Method 500 includes, in step 520, activating the actuator of the robot arm 122 to move the display device 120 to the unobstructed position determined in step 518. The process may continue to verify whether the display device 120 has adapted to the subsequent obstacle in step 510 and whether the obstacle has been successfully removed from the display device 120 by the movement determined in step 518.

[0057] Figure 7 shows an exemplary computing system 700 that implements a controller for executing the control algorithm 400. The ophthalmic microscope 102 and the display device 120 may similarly incorporate computing devices having some or all of the characteristics of the computing system 700.

[0058] As shown in the figure, the computing system 700 includes a central processing unit (CPU) 702 (and possibly a graphics processing unit (GPU)), one or more I / O device interfaces 704 that may allow various I / O devices 714 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to the computing system 700, a network interface 706 to which the computing system 700 is connected to a network 790, memory 708, storage 710, and wiring 712.

[0059] The CPU 702 can retrieve and execute programming instructions stored in memory 708. Similarly, the CPU 702 can retrieve and store application data present in memory 708. Wiring 712 transmits programming instructions and application data between the CPU 702, the I / O device interface 704, the network interface 706, the memory 708, and the storage 710. The CPU 702 is included to represent a single CPU, a multi-CPU, a single CPU with multiple processing cores, and similar configurations.

[0060] Memory 708 represents volatile memory, such as random access memory, and / or non-volatile memory, such as non-volatile random access memory, phase-change random access memory, or similar. As shown in the figure, memory 708 may store executable code for executing the control algorithm 400, and data used by the control algorithm 400, such as kinematic states 402, 404, and images 716 from some or all of the cameras 212, 214, 216.

[0061] The storage 710 may be non-volatile memory, such as disk drives, solid-state drives, or a collection of storage devices distributed across multiple storage systems. The storage 710 may optionally store information, such as a surgeon profile 412, a treatment plan 414, and kinematic data 718 describing the arms supporting the robotic arm 122 and possibly the ophthalmic microscope 102, enabling the association of multiple points in three-dimensional space with the states of the joints of the robotic arm and possibly the arms supporting the ophthalmic microscope 102.

[0062] Additional considerations This description is provided to enable those skilled in the art to implement the various embodiments described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, modifications may be made in the function and arrangement configuration of the elements described herein without departing from the scope of this disclosure. Various examples may omit, substitute, or add various procedures or components as needed. Also, features described in relation to some examples may be combined with some other examples. For example, an apparatus may be realized or a method may be implemented using several embodiments described herein. Furthermore, the scope of this disclosure is intended to cover such apparatus or method implemented using, in addition to or other structures, functions, or structures and functions, in addition to the various embodiments of this disclosure described herein. It should be understood that any embodiment of this disclosure disclosed herein may be embodied by one or more elements of the claims.

[0063] As used herein, the phrase "at least one of" the list of items refers to any combination of those items, including a single component. For example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination comprising multiple of the same elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0064] As used herein, the term “determine” encompasses a wide range of actions. For example, “determine” may include calculating, computing, processing, deriving, investigating, examining (e.g., examining a table, database, or other data structure), finding out, etc. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. It may also include resolving, selecting, choosing, establishing, etc.

[0065] The methods disclosed herein include one or more steps or actions to achieve the method. The steps and / or actions of the method may be substituted for one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of any particular steps and / or actions may be modified without departing from the claims. Furthermore, the various operations of the methods described above may be carried out by any suitable means capable of performing the corresponding function. The means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the drawings, those operations may have corresponding counterpart means-plus-functional components bearing similar reference numerals.

[0066] The logic blocks, modules, and circuits useful for the various descriptions described in connection with this disclosure may be implemented or run using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices (PLDs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. While a general-purpose processor may be a microprocessor, in alternative forms, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0067] The processing system may implement a bus architecture. The bus may include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the processing system. The bus may connect various circuits, in particular, including processors, machine-readable media, and input / output devices. User interfaces (e.g., keypads, displays, mice, joysticks, etc.) may also be connected to the bus. The bus may also connect various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be further described. The processor may implement one or more general-purpose and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuit components capable of running software. Those skilled in the art will recognize the optimal way to perform the described functions of the processing system, depending on the specific application and the overall design constraints imposed on the entire system.

[0068] When executed in software, functions can be stored or transmitted as one or more instructions or code in computer-readable medium. Software will be broadly interpreted to mean instructions, data, or any combination thereof, whether referring to software, firmware, middleware, microcode, hardware description language, or others. Computer-readable medium includes both computer storage media and communication media, such as any media that facilitates the transfer of a computer program from one location to another. A processor may be responsible for managing buses and general-purpose processing, such as the execution of software modules stored in computer-readable storage media. Computer-readable storage media may be coupled to a processor, allowing the processor to read information from and write information to the storage media. In alternative forms, storage media may be essential to the processor. For example, computer-readable medium may include computer-readable storage media storing instructions separated from transmission lines, data-modulated carriers, and / or wireless nodes, all of which may be accessed by the processor through a bus interface. Alternatively, or in addition to, computer-readable medium, or any part thereof, may be incorporated into the processor, such as by caches and / or general-purpose register files. Examples of machine-readable storage media may include, for example, RAM (Random Access Memory), flash memory, ROM (Read-Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage media, or any combination thereof. Machine-readable media can be embodied in computer program products.

[0069] A software module may contain a single instruction or many instructions and may be distributed across several different code segments, within different programs, and across multiple storage media. A computer-readable medium may contain several software modules. A software module contains instructions that, when executed by a device such as a processor, cause a processing system to perform various functions. A software module may include transmit modules and receive modules. Each software module may reside on a single storage device or be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. While a software module is executing, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to the functionality of a software module, it will be understood that such functionality is performed by the processor when instructions from that software module are executed.

[0070] The following claims are not intended to be limited to the embodiments described herein, but to be in full scope consistent with the language of the claims. In the claims, references to singular elements are not intended to mean “one and only one” unless specifically stated so, but rather “one or more.” Unless specifically stated, the term “several” means one or more. The claimed elements shall not be construed under 35 U.S.C § 112(f) unless they are explicitly enumerated using the phrase “means for” or, in the case of a method claim, using the phrase “steps for.” All structural and functional equivalents of the various aspects of the elements described throughout this disclosure, which are known to or will be known to those skilled in the art, are expressly incorporated into this specification and are intended to be included in the claims. Furthermore, nothing disclosed herein is intended to be made available to the public, whether such disclosure is expressly enumerated in the claims or not.

[0071] Exemplary Embodiments Embodiment 1: A system comprising: a display device; an operating structure mounted on the display device and configured to move the display device in three-dimensional space; an imaging device configured to capture a surgical field image of a patient's eye during ophthalmic surgery and display the surgical field image on the display device; a camera mounted on the imaging device; and a controller configured to receive a camera image from the camera; evaluate the representation of the display device in the camera image; and, based on the representation of the display device, activate the operating structure to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery without obstruction.

[0072] Embodiment 2: The system according to Embodiment 1, wherein the controller is configured to: receive a treatment plan for ophthalmic surgery; select the position and orientation of the display device according to the treatment plan; and activate the actuation structure to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery by activating the actuation structure to achieve the position and orientation.

[0073] Embodiment 3: The system according to Embodiment 2, wherein the treatment plan determines the surgeon's position selected from the supra, left temporal, and right temporal regions; and the controller is further configured to select the position and orientation of the display device so that the display device is visible from the surgeon's position in the treatment plan.

[0074] Embodiment 4: The system according to Embodiment 3, wherein the controller is further configured to select the height of the display device according to the surgeon's surgical profile.

[0075] Embodiment 5: The system according to Embodiment 1, wherein the operating structure is a robotic arm.

[0076] Embodiment 6: The system according to Embodiment 1, wherein the operating structure is a robot arm having at least 5 degrees of freedom.

Claims

1. Display device and; An operating structure attached to the display device and configured to move the display device in three-dimensional space; An imaging device configured to capture an image of a patient's eye during ophthalmic surgery and display the image on the display device; A controller configured to operate the operating structure in order to keep the display device within the field of view of the surgeon performing the ophthalmic surgery, A system that includes this.

2. The system according to claim 1, wherein the imaging device is an ophthalmic microscope.

3. The system according to claim 1, wherein the imaging device is an ophthalmic stereomicroscope.

4. The system according to claim 3, wherein the display device is a three-dimensional display device, and the controller is configured to operate the operating structure to maintain the orientation and distance of the display device to the surgeon so as to enable the surgeon to perceive three dimensions.

5. The aforementioned controller is: To receive a surgeon profile; Selecting the position and orientation of the display device according to the surgeon profile; and To activate the operating structure in order to achieve the aforementioned position and orientation. The system according to claim 1, wherein the operating structure is configured to operate in order to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery.

6. The aforementioned controller is: To receive the treatment plan for the aforementioned ophthalmic surgery; Selecting the position and orientation of the display device in accordance with the aforementioned treatment plan; and To activate the operating structure in order to achieve the aforementioned position and orientation. The system according to claim 1, wherein the operating structure is configured to operate in order to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery.

7. The aforementioned treatment plan determines the position of the surgeon selected from the superior, left temporal, and right temporal regions; and The system according to claim 6, wherein the controller is further configured to select the position and orientation of the display device such that the display device is visible from the surgeon's position in the treatment plan.

8. The system according to claim 7, wherein the controller is further configured to select the height of the display device according to the surgeon's surgical profile.

9. The system according to claim 1, wherein the operating structure is a robot arm having at least five degrees of freedom.

10. The aforementioned controller is: Receiving one or more inputs, the inputs being: One or more first camera images received from one or more first cameras mounted on the imaging device; One or more second camera images received from one or more second cameras mounted on the display device; One or more third camera images received from one or more second cameras installed in the operating room, including the imaging device and the display device; First kinematic data of the operating structure; or Second kinematic data of the support for the imaging device Including at least one of the following; and Selecting the position and orientation of the display device based on one or more of the above inputs; and The operating structure is operated to move the display device to the aforementioned position and orientation. The system according to claim 1, wherein the operating structure is configured to operate in order to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery.

11. Display device and; An operating structure attached to the display device and configured to move the display device in three-dimensional space; An imaging device configured to capture an image of the surgical field of a patient's eye during ophthalmic surgery and display the surgical field image on the display device; The camera attached to the aforementioned imaging device; It is a controller, Receive camera images from the aforementioned camera; Evaluate the representation of the display device in the camera image; and Based on the representation of the display device, the operating structure is activated to maintain the display device within the field of view of the surgeon performing the ophthalmic surgery without obstruction. A controller configured in such a way A system that includes this.

12. The system according to claim 11, wherein the imaging device is an ophthalmic microscope.

13. The system according to claim 11, wherein the imaging device is an ophthalmic stereomicroscope.

14. The system according to claim 13, wherein the display device is a three-dimensional display device, and the controller is configured to operate the operating structure to maintain the orientation and distance of the display device to the surgeon so as to enable the surgeon to perceive three dimensions.

15. The aforementioned controller further: To receive a surgeon profile; Selecting the position and orientation of the display device according to the surgeon profile; and To activate the operating structure in order to achieve the aforementioned position and orientation. The system according to claim 11, wherein the operating structure is configured to operate in order to maintain visibility of the display device to the surgeon performing the ophthalmic surgery.

Images