Surgical robotic systems and methods for displaying delay growth

Abstract

A surgical robotic system includes a surgical robot arm having an instrument and an instrument drive unit configured to actuate the instrument; and a surgeon console configured to receive user input to control at least one of the surgical robot arm or the instrument. The system also includes a camera configured to capture the video feed, and a video processing device configured to receive the video feed, calculate a delay of the video feed, and display the delay of the video feed.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 433,502, filed on Dec. 19, 2022, the entire content of which is incorporated herein by reference. BACKGROUND OF THE INVENTION

[0002] Surgical robotic systems are currently used in a variety of surgical procedures, including minimally invasive medical procedures. Some surgical robotic systems include a surgeon's console that controls the surgical arm and a surgical instrument having an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the arm. In operation, the arm moves to a position above the patient and then guides the surgical instrument through a surgical port or natural orifice of the patient into a small incision to position the end effector at a working site within the patient.

[0003] Laparoscopic surgery and robotic surgery rely on real-time visualization. The surgical site is displayed on a monitor within a short delay of up to about 80 milliseconds (ms) between the time the surgical site is captured by a camera and the time it is output on the monitor. This allows the surgeon to view the surgical site almost in real time while performing the surgical procedure, such that there is little to no appreciable time delay between the surgeon's input and the resulting movement of the instrument at the surgical site. Occasionally, due to visual enhancements (e.g., artificial intelligence (AI) overlays, near-infrared imaging, and other techniques in addition to white light laparoscopic imaging), the delay may increase. The increase in delay may be unacceptable due to the time delay between user input (e.g., moving the instrument) and the movement of the instrument displayed on the screen. Therefore, there is a need to alert the user of the increase in delay. SUMMARY OF THE INVENTION

[0004] Delays in surgical visualization can be caused by various factors, including but not limited to video processing latency, transmission latency, AI processing, near-infrared (NIR) imaging, etc. Thus, the delay may fluctuate during the use of the visualization system, depending on various processing enhancements.

[0005] The present disclosure provides a surgical robotic system and method for alerting a user of latency growth. When long latency is encountered, the system also outputs a question asking the user if they want to continue operating the system, thus providing the user with the opportunity to stop or continue operating (e.g., temporarily) despite the latency. After dropping below a preset threshold, the user can be prompted to resume operation. The latency threshold can be preset automatically in the system or adjustable by the user. In embodiments, depending on the type of visualization, there can be different types of thresholds, such as white light imaging, NIR imaging using a dual sensor (e.g., a white light sensor and an NIR sensor), etc. White light imaging can have a latency threshold of about 145 ms, while NIR enhanced imaging can have a latency threshold of about 165 ms to account for processing NIR light images in addition to white light images. The user can adjust these thresholds based on their tolerance for latency with respect to these thresholds.

[0006] According to one embodiment of the present disclosure, a surgical robotic system is disclosed. The surgical robotic system includes: a surgical robotic arm having an instrument and an instrument drive unit configured to actuate the instrument; and a surgeon console configured to receive user input to control at least one of the surgical robotic arm or the instrument. The system further includes a camera and a video processing device, the camera being configured to capture a video feed, and the video processing device being configured to receive the video feed, calculate the latency of the video feed, and display the calculated latency of the video feed.

[0007] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the video processing device may be further configured to compare the latency of the video feed with at least one threshold. The video processing device may be further configured to output an indication in response to the latency of the video feed exceeding at least one threshold. The indication may include at least one of a color-coded latency number, a color-coded frame, or a color-coded video feed coloring. The video processing device is further configured to output a prompt asking whether to continue or pause operating the surgical robotic system in response to the latency of the video feed exceeding at least one threshold. The camera may be configured to capture white light and near-infrared (NIR) light, and the video processing device may be configured to perform at least one of the following: white light imaging, or a combination of white light imaging and NIR light imaging. At least one threshold may be a user-adjustable value. At least one threshold may include a first threshold for white light imaging and a second threshold for a combination of white light imaging and NIR light imaging. The first threshold may be lower than the second threshold. The first threshold may be 145 milliseconds. The second threshold may be 165 milliseconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0009] Figure 1 is a schematic illustration of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system including a control tower, a console, and one or more surgical robotic arms, each of the one or more surgical robotic arms being disposed on a mobile trolley;

[0010] Figure 2 is according to an embodiment of the present disclosure Figure 1 a perspective view of a surgical robotic arm of a surgical robot system;

[0011] Figure 3 is a perspective view of a mobile trolley with an installation arm according to an embodiment of the present disclosure, the mobile trolley carrying Figure 1 a surgical robotic arm of a surgical robot system;

[0012] Figure 4 is according to an embodiment of the present disclosure Figure 1 a schematic diagram of the computer architecture of a surgical robot system;

[0013] Figure 5 is according to an embodiment of the present disclosure Figure 1 a plan view of a surgical robot system, the surgical robot system being positioned around an operating table;

[0014] Figure 6 is a view of the screen of a surgeon's console according to an embodiment of the present disclosure, the screen displaying a recommendation for increased latency; and

[0015] Figure 7 is a flowchart illustrating a method for displaying a recommendation for increased latency according to an embodiment of the present disclosure. Detailed Description

[0016] Embodiments of the surgical robot system disclosed herein are described in detail with reference to the accompanying drawings, in which like reference numerals represent the same or corresponding elements in each of several views.

[0017] As will be described in detail below, the present disclosure relates to a surgical robotic system that includes a surgeon console, a control tower, and one or more mobile carts having surgical robotic arms coupled to mounting arms. The surgeon console receives user inputs through one or more interface devices, and these user inputs are processed by the control tower into movement commands for moving the surgical robotic arms and the instruments and / or cameras coupled to the surgical robotic arms. Thus, the surgeon console enables remote operation of the surgical arms and the attached instruments / cameras. The surgical robotic arm includes a controller configured to process the movement commands and configured to generate torque commands for enabling one or more actuators of the robotic arm, and the one or more actuators then move the robotic arm in response to the movement commands.

[0018] Reference Figure 1 , the surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including the surgeon console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 to which a surgical instrument 50 is removably coupled. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or any number of robotic arms 40.

[0019] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In an embodiment, the surgical instrument 50 may be configured for open surgical procedures. In additional embodiments, the surgical instrument 50 may be an electrosurgical clamp configured to seal tissue by pressing the tissue between jaw members and applying an electrosurgical current thereto. In yet additional embodiments, the surgical instrument 50 may be a surgical stapler that includes a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners (e.g., staples) and cutting the stapled tissue. In yet additional embodiments, the surgical instrument 50 may be a surgical clip applicator that includes a pair of jaws configured to apply a surgical clip to tissue.

[0020] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopic endoscope camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be disposed within the control tower 20. The image processing device 56 may be any computing device configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0021] The surgeon console 30 includes a first screen 32 and a second screen 34. The first screen displays a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 disposed on the robotic arm 40, and the second screen displays a user interface for controlling the surgical robot system 10. The first screen 32 and the second screen 34 can be touchscreens that allow for the display of various graphical user inputs.

[0022] The surgeon console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of hand controllers 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon console further includes an armrest 33 that is used to support the clinician's arm when operating the hand controllers 38a and 38b.

[0023] The control tower 20 includes a screen 23 (which can be a touchscreen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arm 40, such as to move the robotic arm 40 and the corresponding surgical instrument 50 based on a set of programmable instructions and / or input commands from the surgeon console 30 in such a way that the robotic arm 40 and the surgical instrument 50 execute a desired sequence of movements in response to inputs from the foot pedal 36 and the hand controllers 38a and 38b. The foot pedal 36 can be used to enable and lock the hand controllers 38a and 38b, reposition the camera movement, and activate / deactivate electrosurgery. In particular, the foot pedal 36 can be used to perform a clutch action on the hand controllers 38a and 38b. Pressing one of the foot pedals 36 initiates the clutch, which disconnects the hand controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0024] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other by using any suitable communication network based on a wired or wireless communication protocol. As used herein, the term "network", whether plural or singular, refers to a data network, including but not limited to the Internet, intranet, wide area network, or local area network, and is not limited to the entire scope of the definition of the communication network covered by this disclosure. Suitable protocols include but are not limited to Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DCCP). Wireless communication can be achieved through one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data from fixed and mobile devices over short lengths of radio waves), creating a personal area network (PAN), (A specification of an advanced communication protocol using small low-power digital radios based on the IEEE 802.15.4-2003 wireless personal area network (WPAN) standard).

[0025] The computers 21, 31, 41 can include any suitable processor (not shown) that is operably connected to a memory (not shown), which can include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor can be any suitable processor (e.g., control circuit) adapted to execute the operations, calculations, and / or instruction sets described in this disclosure, including but not limited to a hardware processor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will understand that a processor can be replaced by using any logical processor (e.g., control circuit) adapted to execute the algorithms, calculations, and / or instruction sets described herein.

[0026] Reference Figure 2 , each robotic arm 40 can include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. As is known to those skilled in the art, other configurations of links and joints can be used. Joint 44a is configured to fix the robotic arm 40 to the mobile cart 60 and defines a first longitudinal axis. Reference Figure 3, the mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 further includes a screen 69 that is used to display information about the robotic arm 40. In an embodiment, the robotic arm 40 can include any type and / or any number of joints.

[0027] The mounting arm 61 includes a first link 62a, a second link 62b, and a third link 62c that provide lateral maneuverability of the robotic arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint can include an actuator (not shown) that is used to rotate the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c can move in their respective mutually parallel lateral planes, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., an operating table). In an embodiment, the robotic arm 40 can be coupled to an operating table (not shown). The mounting arm 61 includes a control device 65 that is used to adjust the movement of the links 62a, 62b, 62c and the lift 67. In an embodiment, the mounting arm 61 can include any type and / or any number of joints.

[0028] The third link 62c can include a rotatable base 64 that has two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate about a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b can rotate about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for a full three-dimensional orientation of the robotic arm 40.

[0029] The actuator 48b of joint 44b is coupled to joint 44c via belt 45a, and joint 44c is in turn coupled to joint 46b via belt 45b. Joint 44c may include a transfer case that couples belts 45a and 45b such that actuator 48b is configured to rotate each of links 42b, 42c and holder 46 relative to one another. More specifically, links 42b, 42c and holder 46 are passively coupled to actuator 48b, which forces rotation about pivot point "P", which is located at the intersection of a first axis defined by link 42a and a second axis defined by holder 46. In other words, pivot point "P" is the remote center of motion (RCM) of robotic arm 40. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing orientation of surgical instrument 50. Due to the interconnection of links 42a, 42b, 42c and holder 46 via belts 45a and 45b, the angles between links 42a, 42b, 42c and holder 46 are also adjusted to achieve the desired angle θ. In embodiments, some or all of joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0030] Joints 44a and 44b include actuators 48a and 48b that are configured to drive joints 44a, 44b, 44c relative to one another via a series of belts 45a and 45b or other mechanical linkages (such as drive rods, cables, or levers, etc.). In particular, actuator 48a is configured to rotate robotic arm 40 about the longitudinal axis defined by link 42a.

[0031] Reference Figure 2 , holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1 ). IDU 52 is configured to couple to the actuation mechanisms of surgical instrument 50 and camera 51 and is configured to move (e.g., rotate) and actuate instrument 50 and / or camera 51. IDU 52 transfers actuation force from its actuator to components of the end effector 49 of surgical instrument 50. Holder 46 includes a sliding mechanism 46a that is configured to move IDU 52 along the second longitudinal axis defined by holder 46. Holder 46 also includes a joint 46b that rotates holder 46 relative to link 42c. During an endoscopic procedure, instrument 50 may be inserted through an endoscope access port 55 ( Figure 3 ) held by holder 46. Holder 46 also includes a port latch 46c ( Figure 2 ) for securing access port 55 to holder 46.

[0032] The robotic arm 40 further includes a mounting arm 61 and a plurality of manual override buttons 53 disposed on the IDU 52( Figure 1 ), and the mounting arm can be used in the manual mode. The user can press one or more of these buttons 53 to move the components associated with the buttons 53.

[0033] Reference Figure 4 , each of the computers 21, 31, 41 of the surgical robot system 10 may include a plurality of controllers, and the plurality of controllers may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety monitor 21b. The controller 21a receives data from the computer 31 of the surgeon console 30 regarding the current position and / or orientation of the hand controllers 38a and 38b and the status of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine the force feedback commands, which are transmitted back to the computer 31 of the surgeon console 30 to provide haptic feedback through the hand controllers 38a and 38b. The safety monitor 21b checks the validity of the data entering and exiting the controller 21a, and if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or the surgical robot system 10 in a safe state.

[0034] The controller 21a is coupled to a storage device 22a, which may be a non-transitory computer-readable medium configured to store any suitable computer data, such as software instructions executable by the controller 21a. The controller 21a also includes a transient memory 22b for loading instructions and other computer-readable data during the execution of instructions. In an embodiment, the other controllers of the system 10 include a similar configuration.

[0035] The computer 41 includes a plurality of controllers, namely, a cart main controller 41a, a mounting arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart main controller 41a receives and processes the joint commands from the controller 21a of the computer 21, and transmits them to the mounting arm controller 41b, the robotic arm controller 41c, and the IDU controller 41d. The cart main controller 41a also manages instrument replacement and the overall status of moving the cart 60, the robotic arm 40, and the IDU 52. The cart main controller 41a also transmits the actual joint angles back to the controller 21a.

[0036] Each of joints 63a and 63b and the rotatable base 64 of the mounting arm 61 is a passive joint (i.e., in which there is no actuator) that allows the user to manually adjust. Joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the mounting arm 61. When the brakes are engaged, the mounting arm controller 41b monitors the slippage of each of joints 63a and 63b and the rotatable base 64 of the mounting arm 61, or when the brakes are disengaged, the mounting arm can be freely moved by the operator without affecting the control of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates the desired motor torques required for gravity compensation, friction compensation, and closed-loop position control of the robotic arm 40. The robotic arm controller 41c calculates a movement command based on the calculated torques. The calculated motor commands are then transmitted to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted back to the robotic arm controller 41c by the actuators 48a and 48b.

[0037] The IDU controller 41d receives the desired joint angles (such as wrist angle and jaw angle) of the surgical instrument 50 and calculates the desired current of the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart main controller 41a.

[0038] The robotic arm 40 is controlled in response to the pose of a hand controller (e.g., hand controller 38a) that controls the robotic arm 40, and the pose is transformed into the desired pose of the robotic arm 40 by the hand-eye transformation function performed by the controller 21a. The hand-eye function and other functions described herein are implemented in software executable by the controller 21a or any other suitable controller described herein. The pose of one of the hand controllers 38a can be implemented as coordinate positions and roll-pitch-yaw (RPY) orientations relative to a coordinate reference system fixed to the surgeon's console 30. The desired pose of the instrument 50 is relative to the fixed system on the robotic arm 40. Then the pose of the hand controller 38a is scaled by the scaling function performed by the controller 21a. In an embodiment, the coordinate positions can be scaled down and the orientations can be scaled up by the scaling function. Additionally, the controller 21a can also perform a clutch function for disengaging the hand controller 38a from the robotic arm 40. In particular, if certain movement limits or other thresholds are exceeded, the controller 21a stops transmitting movement commands from the hand controller 38a to the robotic arm 40 and substantially acts like a virtual clutch mechanism, e.g., limiting the mechanical input from affecting the mechanical output.

[0039] The desired pose of the robotic arm 40 is based on the pose of the hand controller 38a and is then passed through an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the robotic arm 40 that achieve the scaled and conditioned pose input by the hand controller 38a. The calculated angles are then passed to the robotic arm controller 41c, which includes a joint axis controller with a proportional derivative (PD) controller, a friction estimator module, a gravity compensator module, and a bilateral saturation block configured to limit the commanded torques of the motors of the joints 44a, 44b, 44c.

[0040] Reference Figure 5 , the surgical robot system 10 is installed around the operating table 90. The system 10 includes mobile carts 60a-d, which can be numbered from "1" to "4". During installation, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depend on multiple factors, such as the placement of the multiple access ports 55a-d, which in turn depend on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient, and the carts 60a-d are positioned to insert the instruments 50 and the laparoscopic camera 51 into the corresponding ports 55a-d.

[0041] During use, each of the robotic arms 40a-d is attached to one of the access ports 55a-d inserted into the patient by attaching a latch 46c ( Figure 2 ) to the access port 55 ( Figure 3 ). The IDU 52 is attached to the holder 46, and subsequently the SIM 43 is attached to the distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. Then the instrument 50 is inserted through the access port 55 by moving the IDU52 along the holder 46. The SIM 43 includes a plurality of drive shafts configured to transfer the rotation of the respective motors of the IDU 52 to the instrument 50 to actuate the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40, including the IDU 52. The SIM 43 is also configured to secure a sterile drape (not shown) to the IDU 52.

[0042] Reference Figure 6 , the first screen 32 of the surgeon's console 30 includes a GUI 101 that provides a video feed 102 of the camera 51. The video feed 102 is within the field of view of the camera 51 and can show the surgical site, the instrument 50, etc. The video processing device 56 is configured to output the GUI 101, which can be displayed on any screen of the system 10, for example, the second screen 34 of the surgeon's console 30, the screen 23 of the control tower 20, etc.

[0043] The GUI 101 can be implemented in any laparoscopic visualization system, not just in the surgical robot system 10 of the present disclosure. Thus, the GUI 101 can be used to modify any surgical procedure using manual surgical instruments, powered surgical instruments, or robotic surgical instruments by combining the latency tracking provided by the GUI 101 with cues. Accordingly, the video processing device 56, along with the camera 51 which can be any surgical camera (e.g., an open camera, an endoscopic camera, a capsule camera, a laparoscopic camera, etc.), can be used in any surgical environment with real-time visualization.

[0044] Reference Figure 7 , a method for tracking and displaying latency in a surgical visualization system includes setting a maximum latency for the visualization system, which includes the video processing device 56 and the camera 51. The method can be implemented as software instructions stored in a non-transitory medium (e.g., memory) of the video processing device 56 and executable by one or more processors (e.g., FPGA, CPU, GPU, etc.) of the video processing device 56.

[0045] The camera 51 can be a dual-sensor camera capable of imaging white light and NIR imaging using various contrast agents. In the case of using fluorophores from fluorescent dyes (such as indocyanine green (ICG)) during surgery, the imaging system is capable of performing real-time visual assessment during the surgical procedure and performing real-time visual assessment of blood vessels, lymph nodes, lymphatic flow, bile ducts, and other tissues. The video processing device 56 is configured to combine the white light image and the IR image from the camera 51 by displaying the reflected NIR light as a visible color (e.g., green, blue, etc.) on the video feed 102.

[0046] In step 200, one or more latency thresholds are set, which can be done by the user before or during the procedure. In an embodiment, the latency threshold can be set by the system 10, i.e., set as a default parameter. The video processing device 56 can include thresholds for different types of imaging being performed. Thus, for white light imaging, the video processing device 56 can have a first threshold, which can be from about 100 ms to about 150 ms and can be about 145 ms in an embodiment. For NIR imaging, due to the increased latency in combining the white image and the NIR image, the video processing device 56 can have a second threshold, which can be from about 120 ms to about 180 ms and can be about 165 ms in an embodiment. The thresholds can also be dynamically adjusted based on the imaging mode being used by the video processing device 56. Thus, as different modes are enabled or disabled, the video processing device 56 selects the corresponding threshold.

[0047] In step 202, the video processing device 56 measures the latency of the video feed. This can be done continuously or periodically at any suitable frequency (60 Hz) during the processing of the video feed 102. Any suitable technique for measuring the latency of the video feed can be used, such as timestamps, round-trip time, etc.

[0048] In step 204, the video processing device 56 compares the measured latency with the threshold latency set previously in step 200. This can also be done continuously or periodically at any suitable frequency (60 Hz) during the processing of the video feed 102. The measured latency can also be displayed as a number 104 on the GUI 101. The number 104 can be color-coded to indicate the current latency; thus, a short latency can be colored green, a medium latency can be yellow, and a long latency (i.e., above or near the threshold) can be red. An additional indication of a long latency can be done by using a color-coded frame 106 around the video feed 102. In an embodiment, the video feed 102 can be colored with the same or a similar color-coding to indicate the growth of the latency. Thus, a short latency can be represented as uncolored, and a long latency can be represented by a red coloring of the video feed 102. In addition to the number 104, the color-coding of the frame 106, and / or the video feed 102, the latency number and status can also be displayed in the system message area 108 along with other status messages when these other status messages appear.

[0049] If the latency is below the threshold, the method returns to step 202 to measure the latency and compare the latency at step 204. However, if the latency is above the threshold, then in step 206, the video processing device 56 displays a warning that the latency has exceeded the threshold. This can be done through the message area 108.

[0050] In an embodiment, a prompt notifying the user of a long latency can be displayed on the video feed 102. In step 208, the prompt can also request input from the user. The input can be a response to a question asking the user whether to continue the current operation or stop or pause the program. If the user answers yes, then the system 10 continues to operate, and the video processing device 56 returns to step 202 to measure the latency and compare the latency at step 204. If the user answers no, then the system 10 can stop or pause the operation at step 210. This can be done for a period of time until the latency of the video feed drops below the threshold, and / or this can be done for a predetermined period of time, which can be from about 10 seconds to about 5 minutes. In an embodiment, the system 10 can pause or stop indefinitely until the user actively resumes the operation of the system 10 through the surgeon console 30.

[0051] It will be understood that various modifications may be made to the embodiments disclosed herein. Accordingly, the above description should not be construed as restrictive, but merely as illustrative of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A surgical robot system, comprising: A surgical robotic arm, which includes an instrument and an instrument drive unit configured to actuate the instrument; A surgeon console configured to receive user input to control at least one of the surgical robotic arm or the instrument; A camera configured to capture a video feed; And A video processing device configured to: Receive the video feed; Calculate the latency of the video feed; and Display the latency of the video feed.

2. The surgical robot system according to claim 1, wherein, The video processing device is further configured to compare the latency of the video feed with at least one threshold.

3. The surgical robot system according to claim 2, wherein, The video processing device is further configured to output an indication in response to the latency of the video feed exceeding the at least one threshold.

4. The surgical robot system according to claim 3, wherein The indication includes at least one of a color-coded latency number, a color-coded frame, or a color-coded video feed coloring.

5. The surgical robot system according to claim 3, wherein, The video processing device is further configured to output a prompt asking whether to continue or pause operating the surgical robot system in response to the latency exceeding the at least one threshold.

6. The surgical robot system according to claim 2, wherein, The camera is configured to capture white light and near-infrared (NIR) light, and the video processing device is configured to perform at least one of the following: white light imaging, or a combination of white light imaging and NIR light imaging.

7. The surgical robot system according to claim 6, wherein, The at least one threshold is a user-adjustable value.

8. The surgical robot system according to claim 6, wherein, The at least one threshold includes a first threshold for white light imaging and a second threshold for a combination of white light imaging and NIR light imaging.

9. The surgical robot system according to claim 8, wherein, The first threshold is lower than the second threshold.

10. The surgical robot system according to claim 8, wherein, The first threshold is 145 milliseconds.

11. The surgical robot system according to claim 8, wherein, The second threshold is 165 milliseconds.

Images