System and method for stereoscopic visualization in a surgical robot without the need for glass or headgear

Abstract

A system and method are provided that employ a surgical robotic surgeon console having a display for use by a surgeon in remote surgery. The surgical robotic surgeon console includes a console frame and a horizontal member connected to the console frame, the horizontal member extending outwardly from the console frame and positioned above the display. Also provided is a head bar mounted on the horizontal member and configured to support the user's head relative to the display. Also provided is a system for camera control based at least in part on live gaze position information.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 357,968, filed on July 1, 2022, the content of which is incorporated herein by reference.

Background Art

[0002] A surgical robot system enables a surgeon (also described herein as an operator or user) to perform operations using robotic control equipment to carry out tasks and functions during a procedure. The surgeon can view or observe the operation, including using a visualization system to view images from a camera attached to the robotic control instrument and / or showing the patient. Some existing systems for providing visualization of a surgical robot may require an operator (e.g., a surgeon) to use a peer - in style visualization system, a microscope - like display, or a system that requires polarized glasses or other headgear. These existing systems exhibit some drawbacks. For example, the use of a display such as a peer - in system or a microscope can obscure the operator's peripheral vision and visually separate the operator from activities proceeding in the rest of the operating room. Some three - dimensional (3D) displays require the operator to wear polarized glasses or other headgear that can be cumbersome and / or uncomfortable.

[0003] Some 3D displays limit the light that reaches the user's eyes (like polarized sunglasses), potentially reducing effective resolution and hindering the user's ability to clearly view the 3D display. For example, in some 3D displays, the user wears polarized lenses that filter the appropriate image to the appropriate eye of the user, enabling stereoscopic viewing. In some 3D displays that use polarized filtering, the amount of light reaching each of the user's eyes decreases as a result of the filtering. This reduction in light intensity can reduce the effective solubility (MTF) of the overall image for the user. Some surgeons may be forced to dim or turn off the ambient light in the operating room to at least partially compensate for the reduction in the amount of light reaching the surgeon's eyes when using such 3D displays.

[0004] Furthermore, the filtering may not be perfect, and there can be "crosstalk" for some users between the eyes where the image of one eye bleeds into the other, causing visual interference, reducing effective resolution, causing discomfort, and sometimes causing motion sickness.

[0005] In some systems, the operator leans over a microscope-style stereoscopic viewer. This can cause ergonomic discomfort due to the need for the observer to lean over, and can limit the operator's ability to view the surroundings and interact or observe in the operating room. Summary of the Invention Means for Solving the Problems

[0006] In one embodiment, the present disclosure is directed to a surgical robotic surgeon console that includes a console frame, a display mounted on the console frame, a horizontal member connected to the console frame, and a sensor. The horizontal member extends outwardly from the console frame and may be positioned above the display. The display may be an autostereoscopic display. The sensor may be a head tracking sensor or an eye tracking camera. The surgical surgeon console may further include a head bar mounted on the horizontal member, and the head bar is configured to support the operator's head.

[0007] The surgical robotic surgeon console also includes a head bar mounted on the horizontal member, and the head bar is configured to support the operator's head. The head bar may have a recess therein for receiving the user's forehead. The head bar and the horizontal member may have a width configured to maintain at least the lateral peripheral vision of the operator during use.

[0008] The head bar may be movable from an engaged position to a retracted position. When the head bar is in the engaged position, the head bar may support and position the user's head in a position configured to control and operate the surgical robotic system. When the head bar is in the retracted position, the head bar may be at least partially removed from the user's line of sight, whereby the user can view the display through the head bar without obstruction or with limited instructions.

[0009] In some embodiments, the horizontal member is configured to prevent at least a portion of ambient or overhead light from striking the autostereoscopic display. In some embodiments, the surgical robotic surgeon console also includes one or more sensors for sensing the position of the operator's head relative to the head bar. In some embodiments, the surgical robotic surgeon console also includes at least one camera for imaging at least a portion of the user's head to determine the position of the user's head relative to the head bar or relative to the autostereoscopic display.

[0010] In some embodiments, the surgical robotic surgeon console also includes an eye-tracking camera. In some embodiments, any of the one or more contact sensors, at least one camera, and the eye-tracking camera are mounted or incorporated on the autostereoscopic display, the horizontal bar, or the head support. In some embodiments, the surgical robotic surgeon console also includes one or more operator controls for the surgical robotic system. The eye-tracking camera can be used to identify where the user is looking on the display. The surgical robotic surgeon console can estimate the distance between the camera and the target for each portion of the image and automatically focus the camera at that distance.

[0011] In some embodiments, the present disclosure is directed to a surgical visualization system that includes any of the surgical robotic surgeon consoles described herein, a stereoscopic or 3D display, and / or a sheet fixed in a position relative to the surgical robotic surgeon console. In some embodiments, the position of the sheet is adjustable for different physical characteristics of the operator, and the surgical visualization system is configured to be fixed in different positions corresponding to different physical characteristics of the operator.

[0012] The present disclosure is also directed to a method for controlling a surgical robot system. The method may include tracking the movement of one or more eyes of a user using an eye-tracking camera. The method may include receiving live fixation position information from the eye-tracking camera based on the tracking of the movement of one or more eyes of the user. The method may include setting an adjusted autofocus depth of the camera and presenting an image from the camera based at least in part on the adjusted autofocus depth on a display, at least in part based on the live fixation information. In some embodiments, setting the adjusted autofocus depth of the camera includes pairing the live fixation information with live depth map information generated at least in part from camera data. In some embodiments, the method includes storing a display of the live fixation position information, for example, as an indicator for understanding the use of the surgical robot system. In some embodiments, the method may include storing a display of the live fixation position information. The method may include mapping the live fixation position information to information of one or more of a display, a camera, a robotic arm of the surgical robot system, and a patient. The method may include providing data for local or remote collection.

[0013] In some embodiments, the system according to the present disclosure may enable a surgeon controlling a surgical robot to have a stereoscopic view of the surgical field without the need to use additional eyewear or a stereoscopic viewer. In embodiments, the system according to the present disclosure may reduce ergonomic constraints (e.g.,) by providing a stereoscopic view without the need for the surgeon to don a stereoscopic viewer.

[0014] In some embodiments, the techniques disclosed herein can enable a surgeon to operate a surgical robot in a comfortable position and can prevent the surgeon from being interfered with by the surgeon's existing eyewear or being otherwise annoyed by any additional eyewear that could interfere with the surgeon's existing eyewear or cause discomfort to the surgeon. In some embodiments, the techniques disclosed herein can also enable a more immersive experience for the user. In some embodiments, the techniques disclosed herein can create an "open cockpit" experience that allows the surgeon to recognize and continue to engage with what is happening in the operating room, rather than being separated within a "peer-in" style surgical robot console according to certain existing systems.

[0015] These and other features and advantages of the present invention will be more fully understood by reference to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals refer to like elements throughout the various figures. The drawings illustrate the principles of the invention and are not drawn to scale but show relative dimensions.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7A

Figure 7B

Figure 8

[0017] Disclosed herein are devices, systems, and methods for a surgical robotic surgeon console that include an autostereoscopic display and a horizontal positioning member for at least partially controlling the position of a user's head. As used herein, autostereoscopy refers to a method of displaying a stereoscopic image (e.g., an image that appears three-dimensional) without the use of special headgear, glasses, or lenses worn by the user.

[0018] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the subject matter disclosed herein with respect to the systems and methods of the present disclosure and the environments in which such systems and methods may operate or function. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features well known in the art are not described in detail to avoid obscuring the complexity of the subject matter of the present disclosure and to enhance its clarity. Additionally, any examples provided below are merely illustrative and are not to be construed in a limiting fashion, and it will be understood by the inventors that other systems, devices, and / or methods may be used to implement or complete the teachings of the present invention and be considered within the scope of the present invention.

[0019] While some embodiments of the systems and methods are used to be combined with or incorporated into one or more surgical robot systems described herein, some embodiments may be used in connection with any type of surgical system, including, for example, other types of robotic surgical systems, linear stick type surgical systems, and laparoscopic systems. Further, some embodiments may be combined with, used as part of, or used in other non-surgical systems where a user requires access to a myriad of information while controlling the device or equipment, or in other non-surgical methods.

[0020] In some embodiments, the autostereoscopic visualization system includes a display that can be mounted on a surgeon console for a surgical robotic system, a horizontal or substantially horizontal member, and a head bar or head support mounted on the horizontal member, where the head bar or head support is configured to ergonomically support the user's head when the user is seated in a viewing position at the console. In some embodiments, the horizontal member may be substantially horizontal, for example, angled in an upward or downward direction or arcuate to provide support for the head bar described herein. In some embodiments, the horizontal member is configured with respect to the display to block light mounted on the surrounding or ceiling from casting unwanted shadows or reflections on the display. In some embodiments, the head bar may be configured to guide or enable the surgeon to quickly and easily find an appropriate place to view the display. The display may be a 3D display or a stereoscopic display. In some embodiments, both the horizontal member and the head bar are configured to avoid covering or reducing obstacles in the user's peripheral vision and leaving the user's open peripheral vision so that the user can observe and interact with the rest of the environment (e.g., the operating room). In some embodiments, the surgeon console may be located in a different room and / or location from that of the operating room. In this case, the surgeon may not be able to see the operating room with peripheral vision, but can see the room in which the surgical robotic surgeon console is located with peripheral vision.

[0021] In some embodiments, the headbar includes an ergonomically contoured surface that comfortably fits against the user's forehead. In some embodiments, the headbar is adjustable in the input / output direction, up / down direction, or both, to accommodate different ergonomic requirements of users of different sizes. In some embodiments, the headbar includes a rigid plastic or other similarly rigid material. In some embodiments, the headbar includes a soft material such as rubber or foam configured to conform to the user's head. In some embodiments, the headbar is configured to contact the user's jaw. In some embodiments, the headbar is configured to contact the user's jaw. In some embodiments, the headbar includes an oval surface having either one unified hole or two separate holes (e.g., one for each eye) configured such that when the user positions their head relative to the eye surface, the eye surface surrounds the user's eyes. In this configuration, the holes are large enough not to obscure the user's peripheral vision. In other embodiments, the headbar may consist of a yoke-shaped surface.

[0022] In some embodiments, the headbar is removably attached such that the user has the option of using the visualization system with or without the headbar. In some embodiments, the headbar is stowable or foldable such that the system has at least two configurations, one in which the headbar is actively used and one in which the user can view the stereoscopic image without the assistance of the headbar.

[0023] In some embodiments, the headbar has contacts or at least one sensor for determining when the user's head is present. In some embodiments, the contact or at least one sensor determines when the user's head is in contact with the headbar (e.g., mechanical contact, electrical contact, capacitance sensor, etc.). In some embodiments, the at least one sensor includes one or more sets of at least one laser or light-emitting diode and at least one photodiode configured to detect when a beam of light (e.g., a laser beam) is interrupted by the user's head, and thus whether the user's head is present.

[0024] In some embodiments, imaging may be provided by at least one imaging device (e.g., a video camera). In some embodiments, the imaging device may be mounted on or above a display or horizontal member. In some embodiments, the imaging device may be incorporated into a display or horizontal member. In some embodiments, the presence of the user may be detected using an eye-tracking camera installed, mounted, or incorporated on or above a display or horizontal member. These contacts, sensors, and / or imaging devices may also be used to identify the position of the user, and then the system may compare the position of the user to a desired or appropriate position determined based on a predetermined position or specific characteristics of the user that enables stereoscopic viewing by the user. For example, the eye tracker may be an eye-tracking unit available from Tobi AB (Stockholm, Sweden) or another eye-tracking unit known to those skilled in the art.

[0025] In some embodiments, the headbar and the system enable the user to control the movement of one or more cameras that provide imaging data for the display using sensors on the headbar, such as sensors that measure the pressure applied to the headbar by the user's head. In some embodiments, the detection of the user using the sensor or imaging device provides a trigger to start the display. In some embodiments, the surgical robotic surgeon console or display is activated by user recognition, for example, via visual identification of the user, user authentication by the user entering an access code, or by connecting a hardware key.

[0026] The surgical robotic surgeon console may include a safety controller for providing an alarm in response to receiving a signal from one or more sensors, at least one camera, or an eye-tracking camera that indicates the absence or drowsiness of the user. The alarm may include an audible alarm, a visual alarm, an alarm on display, or a stop.

[0027] Some embodiments provide a stereoscopic display system that includes a stereoscopic display device and a surgical robotic surgeon console configured to generate a stereoscopic image for the user's eyes in an ergonomically comfortable seating or standing position. In some embodiments, the system may include a seat that is integrated with or positioned proximate to the surgical robotic surgeon console and configured to establish a desired spatial relationship between the user seated on the seat and the display of the surgical robotic surgeon console. The desired spatial relationship may be the position of the user relative to the display device that provides visual recognition of the stereoscopic display. In some embodiments, the system may not have a headbar. In these embodiments, the user does not need to contact their head with the surgical robotic surgeon console and can simply position their head so that they can view the display. In some embodiments, a guide may be output to the display to assist the user in positioning their head for optimal viewing of the display.

[0028] In some embodiments, a stereoscopic user interface computer (SUIC) may generate an image to be sent to a display. An eye-tracking camera may measure the distance and direction a user needs to move to align with the display. Next, the SUIC may generate guidance graphics based on that offset data and include them in the image output to the display. The guides may be output on the display or on a separate display or display. The guides may be output via visual or audible means including a horizontal member including a visual indicator or LED indicator on the console. The guides may be an image of the user taken by a camera attached to a surgical robotic surgeon console or horizontal member overlaid with markings to assist the user in moving their head to an optimal position, such as a contour corresponding to the head in an optimal position, markings indicating positions to move the head, or other marks.

[0029] In some embodiments, the display is a lenticular style 3D display technology that does not require eyewear or headgear. In other embodiments, an autostereoscopic display includes a projector-based technology that directs two sides of a stereoscopic image towards and focuses on two eyes of a user separated by the user's interpupillary distance (IPD) with parabolic mirrors or lenses. An example of such a technology is the DRV or DRV-Z1 screen by Vision Engineering. In some embodiments, the display may include any stereoscopic, 3D, or holographic display that does not require the use of additional headgear or glasses. The display may be a digital stereo 3D full high-resolution viewer. The display may be a display system including a projector, mirrors, and a display capable of projecting an image onto the display in 3D view. The display may provide full high-resolution resolution and excellent subject clarity.

[0030] Embodiments can provide certain advantages. A surgical robotic surgeon console can increase the safety of operation, for example, by providing an alarm, inhibiting the movement of the surgical robotic device, or inhibiting the application of electrosurgical energy when it is determined that the user is not looking at the display and / or is not in proximity to the surgical robotic surgeon console. The surgical robotic surgeon console can monitor the surgeon's drowsiness and, upon identifying drowsiness, can provide an alarm or inhibit the operation.

[0031] Before providing additional specific descriptions of the surgical robotic surgeon console with respect to FIGS. 6 - 10, an exemplary surgical robotic system in which embodiments of the surgical robotic surgeon console can be used will be described below with respect to FIGS. 1 - 5.

[0032] Surgical robotic system One embodiment can be used with a surgical robot system. The system for robotic surgery can include a robot subsystem. The robot subsystem can include at least a portion that may also be referred to herein as a robot assembly that can be inserted into a patient via a trocar through a single incision point or site. The portion inserted into the patient via the trocar is small enough to be deployed in vivo at the surgical site and is sufficiently maneuverable when inserted into the body so as to be able to move within the body to perform various surgical procedures at a plurality of different points or sites. The portion inserted into the body to perform functional tasks may be referred to herein as a surgical robot unit, a surgical robot module, or a robot assembly. The surgical robot unit or surgical robot module can include a plurality of different sub-units or components that can be separately inserted into the trocar. The surgical robot unit, surgical robot module, or robot assembly includes a plurality of separate robot arms that can be deployed within the patient along different or distinct axes. These plurality of separate robot arms may be collectively referred to herein as a robot arm assembly. Further, a surgical camera assembly can also be deployed along a separate axis and can form part of the robot unit 50. The surgical robot unit, surgical robot module, or robot assembly can also include a surgical camera assembly. Thus, the surgical robot unit, surgical robot module, or robot assembly uses a plurality of different components such as a pair of robot arms and a surgical or robotic camera assembly, each of which can be deployed along different axes and can be separately operated, maneuvered, and moved. The robot arms and camera assembly that can be disposed along separate operable axes are referred to herein as a split arm (SA) architecture. The SA architecture simplifies and enhances the efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion site, while also assisting in the deployment of the robotic surgical instruments into a surgically ready state and the subsequent removal of the robotic surgical instruments through the trocar.As an example, a surgical instrument can be inserted through a trocar to access the patient's abdominal cavity and perform surgery in vivo. In some embodiments, various surgical instruments, including but not limited to robotic surgical instruments and other surgical instruments known in the art, may be used or employed.

[0033] The systems, devices, and methods disclosed herein are incorporated in and utilized with, for example, robotic surgical devices and related systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US2020 / 39203, and / or camera assemblies and systems disclosed in U.S. Patent Application Publication No. 2019 / 0076199, and / or systems and methods for exchanging surgical tools in a transplantable surgical robotic system disclosed in PCT Patent Application No. PCT / US2021 / 058820. The entire contents and teachings of the above-mentioned patents, patent applications, and publications are hereby incorporated by reference herein. A robotic surgical unit forming part of the present invention can form part of a robotic surgical system, and this robotic subsystem includes, in some embodiments, a surgeon or workstation including suitable sensors and displays, and a robotic support system (RSS) for interacting with and supporting the robotic subsystem of the present invention and forms part of a robotic surgical system. The robotic subsystem includes, in some embodiments, a motor and a robotic surgical unit including one or more robotic arms and one or more camera assemblies. The robotic arm and the camera assembly can form part of a single support axis robotic system, or part of a split arm architecture robotic system, or have other arrangements. The robotic support system can provide a plurality of degrees of freedom so that the robotic unit can be maneuvered to a single position or multiple different positions within the patient. In one embodiment, the robotic support system can be directly attached to an operating table, or the floor or ceiling in the operating room. In another embodiment, the attachment is achieved by various fastening means including, but not limited to, clamps, screws, or combinations thereof. In other embodiments, the structure may be upright. The robotic support system can attach a motor assembly connected to a robotic surgical unit including a robotic arm and a camera assembly. The motor assembly can include gears, motors, drive trains, electronics, etc. for powering the components of the robotic surgical unit.

[0034] The robotic arm and camera assembly are capable of multiple degrees of freedom of movement. According to some embodiments, when the robotic arm and camera assembly are inserted into a patient through a trocar, they are capable of moving in at least the axial, yaw, pitch, and roll directions. The robotic arm incorporates and is designed to utilize a multi-degree-of-freedom robotic arm having an end effector attached to its distal end corresponding to the user's wrist area or joint. In other embodiments, the working end of the robotic arm (e.g., the end effector end) is designed to incorporate, use, or employ other robotic surgical instruments, such as the surgical instrument described in U.S. Patent Application Publication No. 2018 / 0221102, the entire contents of which are incorporated herein by reference.

[0035] Referring to the figures, FIG. 1 is a schematic diagram of a surgical robotic system 10 in which aspects of the present disclosure may be employed according to some embodiments of the present disclosure. The surgical robotic system 10 includes an operator console 11 and a robotic subsystem 20 according to some embodiments.

[0036] The operator console 11 includes a visualization system 9 with a display device 12, an image computer 14 (which may be a three-dimensional (3D) computing module), a hand controller 17 having a sensor and a tracker 16, and a computer 18. Further, the operator console 11 may include a foot pedal array 19 including a plurality of pedals. The foot pedal array 19 includes a sensor transmitter 19A and a sensor receiver 19B and can sense the presence of the user's foot in proximity to the foot pedal array 19.

[0037] The display 12 can be any selected type of display for displaying information, images or videos generated by the image computer 14, the computer 18, and / or the robot subsystem 20. The visualization system 9 can include, for example, a head-mounted display (HMD), an augmented reality (AR) display (e.g., an AR display, or AR glasses combined with a screen or display), a screen or display, a two-dimensional (2D) screen or display, a three-dimensional (3D) screen or display, etc., or can form a part thereof. The visualization system 9 can also include any sensors and trackers 16A. In some embodiments, the display 12 can include an image display for outputting images from the camera assembly 44 of the robot subsystem 20.

[0038] In some embodiments, when the visualization system 9 includes an HMD device, an AR device that senses the head position, or another device using an associated sensing and tracking unit 16A, the HMD device or the head tracking device generates tracking and position data 34A that is received and processed by the image calculation unit 14. In some embodiments, the HMD, AR device, or other head tracking device can provide an operator (e.g., a surgeon, a nurse, or other suitable medical professional) with a display that is at least partially connected or attached to the operator's head, a lens that enables a field of view focused on the display, and a sensing and tracking unit 16A for providing head position and orientation tracking of the operator. The sensing and tracking unit 16A can include, for example, an accelerometer, a gyroscope, a magnetometer, a motion processor, infrared tracking, eye tracking, computer vision, the emission and sensing of alternating magnetic fields, and any other method for tracking at least one of position and orientation, or any combination thereof. In some embodiments, the HMD or AR device can provide image data from the camera assembly 44 to the operator's right and left eyes. In some embodiments, to maintain the operator's virtual reality experience, the sensing and tracking unit 16A tracks the position and orientation of the operator's head, generates tracking and position data 34A, and then can relay the tracking and position data 34A directly to the image calculation unit 14 and / or the calculation unit 18, or via the image calculation unit 14.

[0039] The hand controller 17 is configured to sense the movement of the operator's hand and / or arm to operate the surgical robot system 10. The hand controller 17 may include a sensing and tracking unit 16, circuitry, and / or other hardware. The sensing and tracking unit 16 may include one or more sensors or detectors that sense the movement of the operator's hand. In some embodiments, one or more sensors or detectors that sense the movement of the operator's hand are disposed within a pair of hand controllers that are gripped or engaged by the operator's hand. In some embodiments, one or more sensors or detectors that sense the movement of the operator's hand are coupled to the operator's hand and / or arm. For example, the sensors of the sensor and tracker 16 may be coupled to areas of the hand and / or arm such as the fingers, wrist area, elbow area, and / or shoulder area. When an HMD is not used, in some embodiments, additional sensors can also be coupled to the operator's head and / or neck area. When the operator uses an HMD, eye, head, and / or neck sensors and related tracking techniques may be incorporated within or used with that HMD device and, as described above, can thus form part of the optional sensor and tracking unit 16A. In some embodiments, the sensing and tracking unit 16 may be external and may be coupled to the hand controller 17 via electrical components and / or wearable hardware. In some embodiments, the optional sensor and tracker 16A may sense and track the movement of one or more of the operator's head, the operator's eyes, or at least a portion of the operator's neck, based at least in part on imaging of the operator, in addition to or instead of sensors attached to the operator's body.

[0040] In some embodiments, the sensing and tracking unit 16 can use sensors connected to the operator's torso or any other body part. In some embodiments, the sensor and tracker 16 can use, in addition to the sensor, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of the magnetometer can reduce sensor drift around the vertical axis. In some embodiments, the sensor and tracker 16 also includes sensors placed within surgical materials such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. In some embodiments, the sensors can be disposed outside the operator, such as at a fixed location in a room such as an operating room. The external sensor 37 can be processed by the computer 18 and thus generate external data 36 that can be used by the surgical robot system 10.

[0041] The sensor generates position and / or orientation data indicating the position and / or orientation of the operator's hand and / or arm. The sensor and tracker 16 and / or 16A can be utilized to control the movement (e.g., changes in position and / or orientation) of the camera assembly 44 and the robotic arm 42 of the robot subsystem 20. The tracking and position data 34 generated by the sensor and tracker 16 can be transmitted to the computing module 18 for processing by at least one processor 22.

[0042] Computer 18 can determine or calculate, from tracking and position data 34 and 34A, the position and / or orientation of the operator's hand or arm, and, in some embodiments, also of a part of the operator's head, and transmit the tracking and position data 34 and 34A to the robot subsystem 20. The tracking and position data 34, 34A can be processed by processor 22 and stored, for example, in storage 24. The tracking and position data 34 and 34A can also be used by controller 26, which can generate control signals in response thereto for controlling the movement of robot arm 42 and / or camera assembly 44. For example, controller 26 can change the position and / or orientation of at least a portion of camera assembly 44, at least a portion of robot arm 42, or both. In some embodiments, controller 26 can also adjust the pan and tilt of camera assembly 44 to follow the movement of the operator's head.

[0043] The robot subsystem 20 can include a robot support system (RSS) 46 having a motor 40 and a trochlear 50 or trochlear mount, a robot arm 42, and a camera assembly 44. The robot arm 42 and the camera assembly 44 can form part of a single support axis robot unit as disclosed and described in U.S. Patent No. 10,285,765, or can form part of a split arm (SA) architecture robot system as disclosed and described in PCT Patent Application No. PCT / US2020 / 039203, both of which are hereby incorporated by reference in their entirety.

[0044] The robotic subsystem 20 can utilize a plurality of different robotic arms that can be deployed along different or separate axes. In some embodiments, a camera assembly 44 that can utilize a plurality of different camera elements can also be deployed along a common separate axis. Thus, the surgical robotic system 10 can utilize a pair of separate robotic arms deployable along different axes and a plurality of different components such as the camera assembly 44. In some embodiments, the robotic arm 42 and the camera assembly 44 are separately operable, maneuverable, and movable. The robotic subsystem 20 including the robotic arm 42 and the camera assembly 44 is disposable along separate operable axes and is referred to herein as the SA architecture. The SA architecture simplifies and enhances the efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion point or site, while also assisting in the deployment of the robotic surgical instruments into a surgically ready state and, as further described below, the subsequent removal of the robotic surgical instruments through the trocar 50.

[0045] The RSS 46 can include a motor 40 and a trocar 50 or a trocar mount. The RSS 46 can further include a support member that supports the motor 40 coupled to its distal end. The motor 40 can be coupled to each of the camera assembly 44 and the robotic arm 42. The support member can be configured and controlled to operate one or more components of the robotic subsystem 20 linearly or in any other selected direction or orientation. In some embodiments, the RSS 46 can be upright. In some embodiments, the RSS 46 can include a motor 40 coupled to the robotic subsystem 20 at one end and to an adjustable support member or element at the opposite end.

[0046] The motor 40 can receive a control signal generated by the controller 26. The motor 40 can include gears, one or more motors, a drive train, electronics, etc. for powering and driving the robotic arm 42 and the camera assembly 44 either individually or together. The motor 40 can also provide mechanical power, power, mechanical communication, and electrical communication to the robotic arm 42, the camera assembly 44, and / or other components of the RSS 46 and the robot subsystem 20. The motor 40 can be controlled by the computer 18. Thus, the motor 40 can generate signals for controlling, for example, one or more motors that can control and drive the robotic arm 42, including the position and orientation of each articulating joint of each arm, and the camera assembly 44. The motor 40 can further provide a translational or linear degree of freedom that is first utilized to insert and remove each component of the robot subsystem 20 through the trocar 50. The motor 40 can also be used to adjust the insertion depth of each robotic arm 42 when inserted through the trocar 50 into the patient 100.

[0047] The trocar 50 can be a medical device that, in some embodiments, consists of a claw (which may be a sharp or bladeless tip of metal or plastic), a cannula (essentially a hollow tube), and a seal. The trocar can be used to place at least a portion of the robotic subsystem 20 within an internal cavity of a subject (e.g., a patient) and can draw gas and / or fluid out of the body cavity. The robotic subsystem 20 can be inserted through the trocar to access the patient's body cavity and perform surgery in vivo. In some embodiments, the robotic subsystem 20 of the present invention can be at least partially supported by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and the camera assembly 44 can be maneuvered within the patient at a single position or multiple different positions. In some embodiments, the robotic arm 42 and the camera assembly 44 can be supported by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and the camera assembly 44 can be maneuvered within the patient at a single position or multiple different positions.

[0048] In some embodiments, the RSS 46 can further include an optional controller for processing input data from one or more of the system components (e.g., the display 12, the sensors and trackers 16, the robotic arm 42, the camera assembly 44, etc.) and for generating control signals in response thereto. The motor 40 can also, in some embodiments, include a storage element for storing data.

[0049] In some embodiments and in some operating modes, the robotic arm 42 can be controlled to follow a scaled-down movement or motion of the operator's arm and / or hand as sensed by associated sensors. The robotic arm 42 includes a first robotic arm including a first end effector having an instrument tip disposed at a distal end of the first robotic arm, and a second robotic arm including a second end effector having an instrument tip disposed at a distal end of the second robotic arm. In some embodiments, the robotic arm 42 can have parts or regions associated with movements that can be associated with a shoulder joint, an elbow joint, and a wrist joint and with movements associated with the operator's fingers. For example, the robotic elbow joint can follow the position and orientation of a human elbow, and the robotic wrist joint can follow the position and orientation of a human wrist. The robotic arm 42 can also have an end region associated therewith, which can, in some embodiments, terminate in an end effector that follows the movement of one or more of the operator's fingers, such as the index finger, when, for example, the user pinches the index finger and thumb together. In some embodiments, the robotic arm 42 may follow the movement of the operator's arm in some control modes, while the virtual chest of the robotic assembly may remain stationary (e.g., in an instrument control mode). In some embodiments, the position and orientation of the operator's torso are subtracted from the position and orientation of the operator's arm and / or hand. This subtraction enables the operator to move the torso without moving the robotic arm. The disclosure control regarding the management of the movement of the individual arms of the robotic arm assembly is provided in International Patent Application Publication Nos. WO 2022 / 094000 A1 and WO 2021 / 231402 A1, each of which is incorporated herein by reference in its entirety.

[0050] The camera assembly 44 is configured to not only provide the operator with image data 48, such as, for example, live video feeds of a surgery or a surgical site, but also to enable the operator to operate and control a camera that forms part of the camera assembly 44. In some embodiments, the camera assembly 44 may include one or more cameras (e.g., a pair of cameras) whose optical axes are axially separated by a selected distance, which is known as the inter-camera distance and provides a stereoscopic view or image of the surgical site. In some embodiments, the operator can control the movement of the camera via the movement of the hand, either via a sensor coupled to the operator's hand or via a hand controller grasped or held by the operator's hand, and thus the operator can obtain a desired view of the surgical site in an intuitive and natural manner. In some embodiments, the operator can additionally control the movement of the camera via the movement of the operator's head. The camera assembly 44 is movable in a plurality of directions with respect to the direction of the field of view, including, for example, the yaw direction, the pitch direction, and the roll direction. In some embodiments, the components of the stereoscopic camera can be configured to provide a natural and comfortable user experience. In some embodiments, the inter-axial distance between the cameras can be modified to adjust the perceived depth of the surgical site by the operator.

[0051] The image or video data 48 generated by the camera assembly 44 can be displayed on the display 12. In embodiments where the display 12 includes an HMD, the display can include an embedded sensor and tracker 16A that obtains raw orientation data for the yaw, pitch, and roll directions of the HMD, as well as position data within the Cartesian space (x, y, z) of the HMD. In some embodiments, position and orientation data regarding the operator's head can be provided via a separate head tracker. In some embodiments, the sensor and tracker 16A can be used to provide supplemental position and orientation tracking data for the display, instead of or in addition to an embedded tracking system of the HMD. In some embodiments, operator head tracking is not used or employed. In some embodiments, the operator's image can be used by the sensor and tracker 16A to track at least a portion of the operator's head.

[0052] Figure 2A shows an exemplary robotic assembly 20 (also referred to herein as a robotic subsystem) of the surgical robot system 10 incorporated into or mounted on a mobile patient cart, according to some embodiments. In some embodiments, the robotic assembly 20 includes an RSS 46, which in turn includes a motor 40, the robotic arm assembly 42 has an end effector 45, the camera assembly 44 has one or more cameras 47, and may also include a trocar 50 or trocar mount.

[0053] Figure 2B shows an example of an operator console 11 of the surgical robot system 10 of the present disclosure, according to some embodiments. The operator console 11 includes a display 12, a hand controller 17, and also includes one or more additional controllers such as a foot pedal array 19 for controlling the robotic arm 42, controlling the camera assembly 44, and controlling other aspects of the system.

[0054] FIG. 2B also illustrates the left hand controller subsystem 23A and the right hand controller subsystem 23B of the operator console. The left hand controller subsystem 23A includes and supports the left hand controller 17A, and the right hand controller subsystem 23B includes and supports the right hand controller 17B. In some embodiments, the left hand controller subsystem 23A may be removably connected or engaged to the left hand controller 17A, and the right hand controller subsystem 23B may be removably connected or engaged to the right hand controller 17A. In some embodiments, the connection may be both physical and electronic such that the left hand controller subsystem 23A and the right hand controller subsystem 23B can receive signals from the left hand controller 17A and the right hand controller 17B respectively, including signals conveying input received from user selections on buttons or touch input devices of the left hand controller 17A or the right hand controller 17B.

[0055] Each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may include components that enable the range of motion of their respective left hand controller 17A and right hand controller 17B, such that the left hand controller 17A and the right hand controller 17B may translate or displace in three dimensions and may additionally move in roll, pitch, and yaw directions. Further, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may register the movement of their respective left hand controller 17A and right hand controller 17B in each of the aforementioned directions, and may transmit signals providing such movement information to a processor (not shown) of the surgical robot system.

[0056] In some embodiments, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may be configured to receive and connect to, or engage with, different hand controllers (not shown). For example, hand controllers having different configurations of buttons and touch input devices may be provided. Further, hand controllers having different shapes may be provided. The hand controllers may be selected to be compatible with a particular surgical robotic system or a particular surgical robotic procedure, or may be selected based on the operator's preference for the buttons and input devices, or with respect to the shape of the hand controller, to provide greater comfort and ease for the operator.

[0057] FIG. 3A schematically shows a side view of a surgical robot system 10 for performing surgery within an internal cavity 104 of a subject 100, according to some embodiments and for some surgical procedures. FIG. 3B schematically shows a perspective top view of the surgical robot system 10 performing surgery within the internal cavity 104 of the subject 100. The robot arm assembly 42 includes a robot arm 42A and a robot arm 42B. The subject 100 (e.g., a patient) is placed on an operating table 102 (e.g., a surgical operating table). In some embodiments and for some surgical procedures, an incision is made in the patient 100 to gain access to the internal cavity 104. Next, a trocar 50 is inserted into the patient 100 at a selected location to provide access to the internal cavity 104 or the surgical site. The RSS 46 can then be maneuvered to a position on the patient 100 and the trocar 50. In some embodiments, the RSS 46 includes a trocar mount that couples to the trocar 50. The robot assembly 20 can be coupled to the motor unit 40, and at least a portion of the robot assembly can be inserted into the trocar 50 and thus into the internal cavity 104 of the subject 100. For example, the camera assembly 44 and the robot arm assembly 42 can be individually and sequentially inserted into the patient 100 through the trocar 50. The camera assembly and the robot arm assembly can include portions that remain outside the subject's body during use, but references to inserting the robot arm assembly 42 and / or the camera assembly into the subject's internal cavity, as well as positioning the robot arm assembly 42 and / or the camera assembly 44 within the subject's internal cavity, refer to the portions of the robot arm assembly 42 and the camera assembly 44 that are intended to be within the subject's internal cavity during use. The sequential insertion method has the advantage of supporting smaller trocars and thus allowing for a smaller incision to be made in the patient 100, thereby reducing the trauma experienced by the patient 100. In some embodiments, the camera assembly 44 and the robot arm assembly 42 can be inserted in any order or a specific order.

[0058] In some embodiments, following the camera assembly 44, the first robotic arm of the robotic arm assembly 42 follows, and then the second robotic arm of the robotic arm assembly 42 follows, all of which can be inserted within the trocar 50 and thus within the internal cavity 104. When inserted into the patient 100, the RSS 46 can move the robotic arm assembly 42 and the camera assembly 44 to the surgical site that is manually or automatically controlled by the operator console 11.

[0059] Disclosure control regarding the management of the movement of the individual arms of the robotic arm assembly is provided in International Patent Application Publication Nos. 2022 / 094000 (A1) and 2021 / 231402 (A1), each of which is hereby incorporated by reference in its entirety.

[0060] FIG. 4A is a perspective view of a robotic arm sub-assembly 21 according to some embodiments. The robotic arm sub-assembly 21 includes a robotic arm 42A, an end effector 45 having an instrument tip 120 (e.g., monopolar forceps, a needle driver / holder, a bipolar grasper, or any other suitable tool), and a shaft 122 that supports the robotic arm 42A. The distal end of the shaft 122 is coupled to the robotic arm 42A, and the proximal end of the shaft 122 is coupled to the housing 124 of the motor 40 (as shown in FIGS. 1 and 2A). At least a portion of the shaft 122 can be external to the internal cavity 104 (as shown in FIGS. 3A and 3B). At least a portion of the shaft 122 can be inserted into the internal cavity 104 (as shown in FIGS. 3A and 3B).

[0061] Figure 4B is a side view of the robotic arm assembly 42. The robotic arm assembly 42 includes, in some embodiments, a virtual shoulder 126, a virtual elbow 128 having a volumetric proximity sensor 132, a virtual wrist 130, and an end effector 45. In some embodiments, the virtual shoulder 126, the virtual elbow 128, and the virtual wrist 130 include a series of hinges and rotational joints to provide seven degrees of positioning freedom for each arm, along with an additional one gripping degree of freedom for the end effector 45.

[0062] Figure 5 illustrates a perspective front view of a portion of the robotic assembly 20 configured for insertion into a patient's internal body cavity. The robotic assembly 20 includes a first robotic arm 42A and a second robotic arm 42B. The two robotic arms 42A and 42B may define a virtual chest 140 of the robotic arm 20 in some embodiments. In some embodiments, the virtual chest 140 may be defined by a chest plane that extends between a first pivot point 142A of the most proximal joint of the robotic arm 42A (e.g., shoulder joint 126), a second pivot point 142B of the most proximal joint of the robotic arm 42B, and a camera imaging center point 144 of the camera 47. The pivot center 146 of the virtual chest 140 is at the center of the virtual chest.

[0063] In some embodiments, sensors on one or both of the first robotic arm 42A and the second robotic arm 42B can be used by the system to determine changes in the position of at least a portion of the robotic arm within a three-dimensional space. In some embodiments, sensors on one or both of the first robotic arm and the second robotic arm can be used by the system to determine the position of at least a portion of one robotic arm relative to the position of at least a portion of the other robotic arm within a three-dimensional space.

[0064] In some embodiments, the camera assembly 44 is configured to acquire an image such that the system can determine the relative position within the three-dimensional space. For example, the camera assembly may include a plurality of cameras, at least two of which are laterally displaced from each other with respect to the imaging axis, and the system may be configured to determine the distance to a feature within the internal body cavity. Further disclosure regarding a camera assembly and associated system for determining the distance to a feature is found in International Patent Application Publication No. 2021 / 159048, entitled "System and Method for Determining Depth Perception In Vivo in a Surgical Robotic System", published on August 12, 2021, which is hereby incorporated by reference in its entirety. Information regarding the distance to a feature of the camera and information regarding the optical characteristics may be used by the system to determine the relative position within the three-dimensional space.

[0065] Surgical robotic surgeon console with a stereoscopic visualization system Also refer to FIGS. 6A - 10 for a stereoscopic or 3D visualization system and method, and a surgical robotic surgeon console using the stereoscopic 3D visualization system and method.

[0066] FIGS. 6A and 6B respectively show a side view and a top view of a surgical robotic surgeon console 200 including a console frame 210, a display 220 mounted on the console frame 210, a horizontal member 230 connected to the console frame 210 and positioned above the display 220, a head bar 240 connected to the horizontal member 230, and a control assembly 270. The user 250 is positioned on a seat 260 such that when the user 250 is positioned at the surgical robotic surgeon console 200, for example, when the user 250 is positioned to view the display 220 and the control assembly 270, the user 250 can hold the head stationary with respect to the head bar 240 or position the head in proximity to the head bar 240.

[0067] The head bar 240 may include one or more head position sensors 247. The head bar 240 may include a head position sensor 247 to determine whether the user 250's head is positioned near, adjacent to, or touching the head bar 240. The head position sensor 247 may be selected from a pressure sensor, a contact sensor, an electrical sensor, a photoelectric sensor, a light beam and detector sensor, a proximity sensor, or a camera. The head position sensor 247 may be incorporated into the shroud of the head bar 240. The head position sensor 247 may identify whether the user 250's head is appropriately spaced from the head bar 240 (e.g., near, in close proximity to, or in contact with the head bar 240). For example, if the head position sensor 247 is a light beam and a corresponding light detector, the user's head may break the light beam and light detector, indicating that the user 250's head is appropriately spaced from the head bar 240. If the head position sensor 247 can sense that the user 250 is in proximity to the head bar 240 without requiring contact between the user 250 and the head bar 240, the user 250 need not place his or her head against the head bar 240. If the surgical robot system 10 identifies, via the head position sensor 247, that the user 250 is not in contact with or in proximity to the head bar 240, the surgical robot system 10 may provide an audible or visual alarm or may disable the operation of one or more portions of the surgical robot system 10.

[0068] The head bar 240 can be operated by the user to tilt the head bar 240 in an upward or downward direction and away from the user 250. The head bar 240 has an engaged position (e.g., FIGS. 6A-6B) and a retracted position (e.g., FIG. 6C). In the engaged position, the head bar 240 can be positioned in a position configured to support the user's head and control and operate the surgical robot system 10 as needed. When the head bar 240 is in the retracted position, the head bar 240 is at least partially removed from the user 250's line of sight, allowing the user 250 to view the display 220 through the head bar 240 without or with limited obstructions.

[0069] The horizontal member 230 is configured to block ambient or ceiling-mounted light, such as ceiling light 310, from casting unwanted shadows, reflections, or glares on the display. Specifically, the ceiling light 310 emits light rays 312 that are at least partially blocked by the horizontal member 230 and strike the display device 220. The horizontal member 230 may also include another light shield, which may be stowable or foldable to further block light from ceiling light or ambient light.

[0070] FIG. 6C shows a side view of a surgical robotic surgeon console 200 showing a console frame 210, a display 220 mounted on the console frame 210, a horizontal member 230 connected to the console frame 210, and a head bar 240. In FIG. 6C, the head bar 240 is tilted vertically from the user 250 in the stowed position. By tilting the head bar 240 upward, the user can maintain their head in substantially the same position without placing it against the head bar 240. The inventors have determined that some users 250 have different preferences regarding the use of the head bar 240. For example, some users 250 may prefer to use the head bar 240 throughout the procedure, some users may prefer not to use the head bar 240, and some users may prefer to use the head bar 240 during one or more parts of the procedure and not use the head bar 240 during one or more different parts of the procedure. Some users 250 have found that the head bar 240 can become uncomfortable after long-term use. The bar 240 in FIG. 6C may be tilted upward away from the user 250's head or pivoted. The pivoting mechanism can be achieved by various features known to those skilled in the art, including but not limited to a hinge, a ball and socket connection, a shaft, or a portion of a flexible material. The head bar 240 may be positioned to have three or more positions, such as three, four, five, or more positions, or an infinite number of positions where the pivoting mechanism provides continuous positions.

[0071] The head bar 240 may be manually operable by the user 250 or may be operated via a mechanical, electrical, or hydraulic system. The head bar 240 may be driven by a motor and controlled by the surgical robotic system 10 to enable movement of the head bar 240.

[0072] Figure 6D shows a surgical robot surgeon console 200 that depicts the peripheral vision 301, 302 of user 250. Figure 6D shows an exemplary range of the peripheral vision 301, 302 of user 250 when user 250 is positioned at the surgical robot surgeon console 200 and their head is placed against or positioned proximate to the head bar 240. Peripheral vision refers to the user's side vision, in other words, the area that appears to the side of the user when the user is looking straight ahead. Peripheral vision consists of that portion of vision other than the center of fixation and can be divided into far peripheral, mid-peripheral, and near-peripheral. The peripheral vision 301, 302 of user 250 can be relevant to the surgical situation such that the user can see instruments, displays, equipment, notes, or people (e.g., surgical staff, physicians, anesthesiologists, technicians, operating room nurses) in the environment of the surgical robot surgeon console 200 while maintaining the center of fixation on the surgical robot surgeon console 200 and display 220 within their peripheral vision 301, 302. By providing an increased peripheral field of view, the user can see more of the environment of the surgical robot surgeon console, which can be the operating room or a remote site hosting the surgical robot surgeon console 200, enabling user 250 during the procedure. According to embodiments disclosed herein, the head bar 240 and the horizontal member 230 can be configured to maximize the range of the peripheral vision 301, 302 of user 250. For example, the horizontal member 230 may be recessed from the position of user 250, and the horizontal member 230 and / or the head bar 240 may be narrow enough to allow the desired range of the peripheral fields of view 301, 302, or both. In Figure 6D, the increased peripheral vision can enable user 250 to interact better by seeing other people and / or objects present in the environment of the surgical robot surgeon console 200 in addition to the surgical robot surgeon console 200.

[0073] Figures 7A and 7B illustrate a side view and a top view, respectively, of a surgical robot surgeon console 300. Features of the surgical robot surgeon console 300 that are similar to those of the surgical robot surgeon console 200 are identified by the same reference numbers for convenience. The surgical robot surgeon console 300 does not include a head bar. The surgical robot surgeon console 300 includes an eye-tracking camera 280. The eye-tracking camera 280 can be used to detect the presence of a user (e.g., user 250) in the surgical robot surgeon console 300. In some embodiments, the eye-tracking camera 280 can be used to identify the position of the user and to confirm that the user is properly positioned relative to the surgical robot surgeon console to operate the control assembly and view the display. Further, the seat 260 of the surgical robot surgeon console 300 can be positioned and configured such that when the user 250 is seated on the seat 260 and properly positioned to operate the surgical robot system 10, the control assembly 270 is positioned such that the user is properly positioned to view the display 220.

[0074] FIG. 8 shows a perspective view of a surgical robot surgeon console 400. Features of the surgical robot surgeon console 400 that are similar to those of the surgical robot surgeon console 200 are identified by the same reference numerals for convenience. The surgical robot surgeon console 400 includes a console frame 210 that supports a control assembly 270 and a display cabinet 221. The display cabinet 221 holds a display system 242 and an eye-tracking camera 280. The display cabinet 221 is connected to a horizontal member 230, which in some embodiments includes a forehead rest 245. The display system 242 includes a projector 244 that projects an image onto a display device 220, a mirror 246 (not shown) located within the forehead rest 245 to reflect the image displayed on the display device 220, and the display device 220. The display cabinet 221 may be adjustable in a vertical dimension relative to the console frame 210 to accommodate the height and preferred position of the user 250 and to further support an ergonomically preferred configuration of the surgical robot surgeon console 400 by enabling adjustment of the height of the display cabinet 221.

[0075] The eye-tracking camera 280 is mounted below the display device 220 within the display cabinet 221. The eye-tracking camera 280 can identify whether the eyes of the user 250 are within the field of view 281 of the eye-tracking camera 280. In some embodiments, the eye-tracking camera 280 can detect when the user 250 focuses on the display 220. The eye-tracking camera 280 can detect when the user 250 is too far away from the control assembly 270 for the control assembly 270 to operate safely. The display device 220 and the horizontal member 230 are tilted upward so that occlusion of the field of view 281 by the forehead rest 245 is minimized.

[0076] The surgical robot surgeon console 400 may inhibit or suspend the operation of the surgical robot system 10 when certain conditions are met based on the input from the eye-tracking camera 280. For example, the surgical robot surgeon console 400 may inhibit or suspend the operation of the surgical robot system 10 when the eye-tracking camera 280 identifies that the user 250 is not looking at the display 220, the user 250 is asleep, the user 250 has been removed too far from the display 220 or the control assembly 270 to safely control the surgical robot system 10, or the user 250 is not present. Inhibiting or suspending the operation of the surgical robot system 10 may include inhibiting the movement of the surgical robot system 10, inhibiting the application of electrosurgical energy, inhibiting other specific functions or the surgical robot system 10, or inhibiting all operations of the surgical robot system 10.

[0077] Camera Control and Live Gaze Position Information Embodiments provide camera control that uses data from the eye-tracking camera 280 to control the camera 47. The eye-tracking camera 280 provides information about the live gaze position of the user 250, i.e., where the user 250 is currently looking with respect to the display 220. The surgical robot surgeon console 200 pairs the live gaze position information with the live depth map information of the scene to set the autofocus depth on the camera 47. The eye-tracking camera 280 may be used to identify where on the display 220 the user 250 is looking.

[0078] The surgical robot surgeon console 200 can estimate the distance between the camera 47 and the target for each part of the image and automatically focus the camera 47 at that distance. When the autofocus depth is set using live gaze information and live depth map information, the depth of focus of the image shown by the display 220 coincides with and is adjusted to the location where the user 250 is looking at the display 240. By automatically adjusting the image (e.g., for automatically adjusting the focus of the camera), the amount and frequency of manual adjustment by the user 250 are reduced or eliminated compared to a fixed image. The resulting image is closer to the image desired by the user 250 than can be achieved by making assumptions about where the user 250 is looking without live gaze position information. The eye tracking camera 280 is also configured to identify when the user 250 is blinking or closing their eyes and to provide information that the user 250 is blinking or closing their eyes so that a direction can be sent by the surgical robot surgeon console 200 to the camera assembly 44 for performing a camera cleaning operation during the time the user 250 is blinking or closing their eyes.

[0079] In some embodiments, the surgical robot surgeon console 200 stores the display of live gaze position information from the eye tracking camera 280. In some embodiments, the surgical robot surgeon console 200 maps the live gaze position information from the eye tracking camera 280 to information on one or more of the display 220, the camera 47, the robotic arm 42 of the surgical robot system 10, and the patient. In some embodiments, the surgical robot surgeon console 200 provides data for local or remote collection. For example, the data may be provided to an offline machine to develop a machine learning system, or for use in a machine learning system, to develop information regarding the user's gaze during the procedure and / or the user's use of images, menus, and other information from the display 220 during the procedure. Incorporating data on one or more of the display 220, the camera 47, the robotic arm 42 of the surgical robot system 10, and the patient, the data from the gaze position information can be differentiated for different parts of the procedure or different tasks performed by the surgeon. In some embodiments, a representation of the live gaze position information is displayed, for example, present in the operating room or displayed to someone other than the user viewing the surgery, such that someone other than the user can know where the user is looking.

[0080] In some embodiments, a guide may be output to the display 220 to assist the user 250 in positioning their head for optimal viewing of the display 220. In some embodiments, the image computer 14 comprises a stereoscopic user interface computer (SUIC) that may generate an image to be transmitted to the display device 220. The eye tracking camera 280 may measure the distance and direction the user 250 needs to move to align with the display. The SUIC may then generate guidance graphics based on that offset data and may include them in the image output to the display 220. The guide may be output on the display 220 or on a separate display or display. The guide may be output via visual or audible means, including visual indicators on the surgical robot surgeon console 300 or on the horizontal member 230 including LED indicators. The guide may overlay a mark to assist the user 250 in moving to an optimal position, such as a contour corresponding to an optimally positioned head, a marking indicating a position to move the head, or other mark, and may show an image of the user taken by a camera mounted on the surgical robot surgeon console 300 or the horizontal member 230.

[0081] Although some embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the present invention. It is understood that various alternatives to the embodiments of the present invention described herein may be used. The following claims define the scope of the present invention, and it is intended that methods and structures within these claims and their equivalents be covered thereby.

Claims

1. A robotic surgeon console for surgery, a console frame, a display mounted on the console frame, a horizontal positioning member connected to the console frame, the horizontal member extending outwardly from the console frame and positioned above the display, and a sensor.

2. The robotic surgeon console according to claim 1, wherein the display is an autostereoscopic display.

3. The robotic surgeon console according to claim 1, wherein the sensor is a head position sensor or an eye tracking camera.

4. The robotic surgeon console according to claim 1, further comprising a head bar mounted on the horizontal member, the head bar being configured to support the user's head.

5. The robotic surgeon console according to claim 4, wherein the head bar includes a recess for receiving the user's forehead therein.

6. The robotic surgeon console according to claim 4, wherein the head bar is movable from an engaged position to a retracted position.

7. The robotic surgeon console according to claim 4, wherein the head bar and the horizontal member have a width configured to maintain at least the user's lateral peripheral vision during use.

8. The robotic surgeon console according to claim 1, wherein the horizontal member is configured to prevent at least a portion of ambient or overhead light from striking the display.

9. The robotic surgeon console according to claim 4, further comprising one or more sensors for sensing the position of the user's head relative to the head bar.

10. The robotic surgeon console according to claim 9, wherein at least one of the one or more sensors is positioned on the head bar.

11. The robotic surgeon console according to claim 10, wherein the at least one of the one or more sensors is selected from a pressure sensor, a contact sensor, an electrical sensor, a photoelectric sensor, a light beam and detector sensor, a proximity sensor, or a camera.

12. The surgical robot surgeon console according to claim 1, further comprising at least one camera for imaging at least a part of the user's head to determine the position of the user's head relative to the head bar or relative to the display.

13. The surgical robot surgeon console according to claim 12, wherein the at least one camera is mounted on or incorporated into the display, the horizontal bar, or the head support.

14. The surgical robot surgeon console according to claim 1, further comprising an eye-tracking camera for tracking the user's live gaze information.

15. The surgical robot surgeon console according to claim 1, further comprising one or more operator controls for a surgical robot system.

16. The surgical robot surgeon console according to claim 15, wherein the alarm includes an audible alarm, a visual alarm, an alarm displayed on the display, or a function stop.

17. The surgical robot surgeon console according to claim 1, further comprising a sheet positionable relative to the display, the console frame, or both.

18. The surgical robot surgeon console according to claim 17, wherein the position of the sheet is adjustable for different body characteristics of the user, and the surgical visualization system is configured such that the sheet is fixed at different positions corresponding to different body characteristics of different users.

19. A surgical robot system, The surgical robot surgeon console according to claim 1, wherein the sensor is an eye-tracking camera, A surgical camera assembly, One or more controllers that receive an input from the eye-tracking camera and control the operation or function of the surgical camera assembly. A surgical robot system comprising:

20. A method of controlling a surgical robot system, Tracking the movement of one or more eyes of a user using an eye-tracking camera; Receiving live gaze position information from the eye-tracking camera based on the tracking of the movement of one or more eyes of the user; Setting an adjusted autofocus depth of the camera based at least in part on the live gaze position information. A method comprising presenting an image from the camera on the display based at least in part on the adjusted autofocus depth. **Claim 21** The method of claim 20, wherein setting the adjusted autofocus depth of the camera includes pairing live gaze information with live depth map information generated at least in part from camera data. **Claim 22** Storing a display of the live gaze position information; Mapping the live gaze position information to information of one or more of the display, the camera, a robotic arm of the surgical robotic system, and the patient; The method of claim 20, further comprising providing data for local or remote collection. **Claim 23** The method of claim 20, wherein the display is an autostereoscopic display.

Images