Surgical robotic system and method for communicating between surgeon console and bedside assistant

Abstract

A surgical robotic system includes: an assistant access port configured to receive an assistant instrument; an endoscope camera configured to generate a video feed of the surgical site and the helper instrument; and a control tower having a first screen; and a surgeon console having a second screen and a handle controller, the surgeon console configured to receive user input. The system further includes a video processing device configured to render a virtual instrument of the assistant instrument in the video feed to generate an enhanced video feed; moving the virtual instrument in the enhanced video feed in response to the received user input; outputting an enhanced video feed with a virtual instrument on the first screen and the second screen; confirming whether the assistant instrument is placed at the position of the virtual instrument; and indicating whether the assistant instrument is placed at the position of the virtual instrument on the first screen and the second screen.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 437,786, filed January 9, 2023, the entire contents of which are incorporated herein by reference. Background Art

[0003] Surgical robotic systems are used in a variety of surgical procedures, including minimally invasive medical procedures. Some surgical robotic systems include a surgeon's console that controls a surgical robotic arm and a surgical instrument with an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the robotic arm. During operation, the robotic arm moves to a position above the patient and then guides the surgical instrument into a small incision through the patient's surgical port or natural orifice to position the end effector at the working site within the patient's body.

[0004] Minimally invasive surgery and robotic-assisted surgery enable surgeons to perform surgical procedures in a teleoperation mode from a remote console. During teleoperation, surgeons scrub and leave the sterile field during the surgical procedure. There are various needs for bedside assistance during surgical procedures. Some assistance activities require streamlined communication between the assistant and the surgeon. There is an unmet need for optimized communication between the assistant and the surgeon. Summary of the Invention

[0005] The present disclosure provides an augmented reality (AR) guided communication platform between a surgeon and an assistant, where the surgeon can show the assistant how to perform complex maneuvers through tool gestures. This avoids the surgeon having to interrupt the surgical workflow or have to scrub in and perform laparoscopic maneuvers himself. Instead, the surgeon can transmit complex surgical maneuvers to the assistant by manipulating a virtual tool, which both the surgeon and assistant can visualize on one or more of their respective screens.

[0006] According to one embodiment of the present disclosure, a surgical robot system is disclosed. The surgical robot system includes an assistant access port configured to receive an assistant instrument. The system also includes an endoscopic camera configured to generate a video feed of the surgical site and the assistant instrument. The system further includes: a control tower having a first screen; and a surgeon's console having a second screen and a handle controller, the surgeon's console configured to receive user input. The system further includes a video processing device configured to render a virtual instrument of the assistant instrument in the video feed to generate an enhanced video feed, and to move the virtual instrument in the enhanced video feed in response to user input. The video processing device is further configured to output the enhanced video feed with the virtual instrument on the first screen and the second screen, and to confirm whether the assistant instrument is placed at the position of the virtual instrument. In addition, the video processing device is configured to indicate on the first screen and the second screen whether the assistant instrument is placed at the position of the virtual instrument.

[0007] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the surgical robot system may further include: a robotic arm having a robotic instrument; and a robotic access port configured to receive a robotic instrument. The surgeon's console may be configured to switch between controlling the robotic instrument and controlling the virtual instrument. The surgical robot system may further include a tracking unit configured to track the position of the assistant's access port and the position of the robot's access port. The video processing device may be configured to determine the positioning of the assistant's instrument based on the position of the assistant's access port and the position of the robot's access port. The video processing device may also be configured to render the virtual instrument based on 3D model data of the assistant's instrument. The endoscopic camera may be a stereo camera and the processing device may be configured to generate a depth map of the surgical site. The processing device may further be configured to generate a virtual boundary corresponding to the physical boundary of the surgical site based on the depth map. The surgeon's console may also be configured to limit user input for controlling the movement of the virtual instrument at the surgeon's console based on the virtual boundary.

[0008] According to another embodiment of the present disclosure, a non-transitory computer-readable medium is disclosed. The medium stores instructions that, when executed by a processor, cause the processor to perform a computer-implemented method for transmitting movement instructions using a virtual instrument. The method includes: receiving a video feed of a surgical site and an assistant instrument; and rendering a virtual instrument of the assistant instrument in the video feed to generate an enhanced video feed that is displayed on a first screen of a control tower and a second screen of a surgeon's console. The method further includes moving the rendered virtual instrument in the video feed in response to an input signal received from the surgeon's console; and outputting the enhanced video feed with the virtual instrument on the first screen and the second screen. In addition, the method includes confirming whether the assistant instrument is placed at the position of the virtual instrument; and indicating on the first screen and the second screen whether the assistant instrument is placed at the position of the virtual instrument.

[0009] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include controlling switching between a robotic instrument connected to a robotic arm and inserted through a robotic access port and a virtual instrument. The method may further include tracking the position of an assistant access port (through which the assistant instrument is inserted) and the position of the robotic access port. The method may also include determining the positioning of the assistant instrument based on the position of the assistant access port and the position of the robotic access port. In addition, the method may include rendering the virtual instrument based on 3D model data of the assistant instrument. The method may also include generating a depth map of the surgical site. The method may further include generating a virtual boundary corresponding to the physical boundary of the surgical site based on the depth map. The method may also include limiting the movement of the virtual instrument at the surgeon's console based on the virtual boundary.

[0010] According to another embodiment of the present disclosure, a surgical robotic system is disclosed. The surgical robotic system includes a robotic arm having a robotic instrument; and a robotic access port configured to receive the robotic instrument. The system also includes an assistant access port configured to receive an assistant instrument. The system further includes an endoscopic camera configured to generate a video feed of the surgical site and the assistant instrument. The system also includes a control tower having a first screen; and a surgeon's console having a second screen and a handle controller configured to receive user input to control the robotic instrument and a virtual instrument. The system further includes a video processing device configured to render a virtual instrument of the assistant instrument in the video feed to generate an augmented video feed, and to move the virtual instrument in the augmented video feed in response to user input. The video processing device is further configured to output the augmented video feed with the virtual instrument on the first and second screens; confirm whether the assistant instrument is placed at the virtual instrument's location; and indicate on the first and second screens whether the assistant instrument is placed at the virtual instrument's location.

[0011] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the endoscopic camera may be a stereoscopic camera and the processing device may be configured to generate a depth map of the surgical site. The processing device may be configured to generate a virtual boundary corresponding to the physical boundary of the surgical site based on the depth map. The surgeon's console may be configured to restrict user input for controlling movement of virtual instruments at the surgeon's console based on the virtual boundary. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0013] Figure 1 is a schematic illustration of a surgical robotic system according to an embodiment of the present disclosure, the surgical robotic system including a control tower, a control console, and one or more surgical robotic arms, the one or more surgical robotic arms being disposed on a mobile cart;

[0014] Figure 2 According to the embodiment of the present disclosure Figure 1 A three-dimensional diagram of a surgical robot arm of a surgical robot system;

[0015] Figure 3 is a perspective view of a mobile cart with mounting arms according to an embodiment of the present disclosure, the mobile cart having Figure 1 The surgical robotic arm of the surgical robotic system;

[0016] Figure 4 According to the embodiment of the present disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system;

[0017] Figure 5 According to the embodiment of the present disclosure Figure 1 A schematic plan view of a surgical robot system positioned around an operating table;

[0018] Figure 6 is an illustration of a graphical user interface displayed on a control tower display and a surgeon console display according to an embodiment of the present disclosure; and

[0019] Figure 7 is a flow chart illustrating a method for providing movement instructions to a bedside assistant according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the surgical robotic system disclosed herein are described in detail with reference to the drawings, in which like reference numerals designate identical or corresponding elements in each of the several views.

[0021] As will be described in detail below, the present disclosure relates to a surgical robotic system that includes a surgeon's console, a control tower, and one or more mobile carts having a surgical robotic arm coupled to a mounting arm. The surgeon's console receives user inputs through one or more interface devices, which are processed by the control tower into movement commands for moving the surgical robotic arm and instruments and / or cameras coupled to the surgical robotic arm. Thus, the surgeon's console enables remote operation of the surgical arm and attached instruments / cameras. The surgical robotic arm includes a controller that is configured to process the movement commands and is configured to generate torque commands for activating one or more actuators of the robotic arm, which in turn move the robotic arm in response to the movement commands.

[0022] The present disclosure provides a surgical robotic system and method for improving communication between surgeons and assistants. The system enables complex instrument manipulation to be transmitted to the assistant without interrupting the surgical flow by taking eyes off the screen of the surgeon's console or having to scrub and complete the task personally. In particular, the system is configured to virtualize one or more surgical instruments that are controlled by the assistant and can be any powered or manual instrument. The guided communication interface is an augmented reality (AR) interface that generates a virtual image of the instrument as an overlay on the endoscopic camera feed.

[0023] The workflow for optimal communication between the surgeon and the assistant can include the assistant inserting the laparoscopic instruments into the surgical site. The surgeon then moves the endoscopic camera to place the assistant's instruments in the camera's field of view. The camera can be a stereo camera configured to enable depth mapping of the surgical site and instruments. The camera is calibrated to enable accurate depth perception. The assistant's instruments are identified by machine vision. The system also includes an external vision system (e.g., one or more cameras or infrared sensors) for determining the port positions of all access ports. This allows the system to determine the position of the assistant's port relative to the endoscope port so that a virtual instrument is generated in the camera feed, which is shown on the screen of the surgeon's console used by the surgeon and on the screen of the control tower used by the assistant.

[0024] The system is also configured to load the 3D model, physics model, and kinematics of the assistant instrument used to render the virtual instrument. The virtual instrument is then rendered as an AR overlay on both screens. The initial placement of the virtual instrument is based on physical modeling of the stereoscopic reconstructed depth map from the stereo endoscope.

[0025] The virtual instrument is controlled through the surgeon's console in the same manner as any actual robotic controller instrument. The surgeon disengages the clutch, takes control of the virtual instrument rendered on the screen, and uses the handle controller to move the virtual instrument. The virtual tool is associated with the assistant port, and the system is configured to realistically move the virtual tool in the video feed of the surgical site based on the kinematics of the assistant instrument. The surgeon's movement trajectory can be recorded so that it can be replayed by the assistant on the screen in the control tower. The assistant then moves the assistant instrument until the assistant instrument is in the same position as the virtual instrument, i.e., the assistant instrument is aligned with the virtual instrument following the trajectory as a guide. The system can verify that the assistant instrument is in the virtual instrument position and is oriented in the same position based on depth mapping and image processing.

[0026] refer to Figure 1 The surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including a surgeon's console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 to which a surgical instrument 50 is removably coupled. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or any number of robotic arms 40.

[0027] The surgical instrument 50 is configured for use during minimally invasive surgery. In an embodiment, the surgical instrument 50 can be configured for open surgery. In another embodiment, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by compressing the tissue between the jaw members and applying an electrosurgical current thereto. In yet another embodiment, the surgical instrument 50 can be a surgical stapler comprising a pair of jaws configured to grasp and clamp tissue, simultaneously deploy a plurality of tissue fasteners (e.g., staples) and cut the stapled tissue. In yet another embodiment, the surgical instrument 50 can be a surgical clip applicator comprising a pair of jaws configured to apply a surgical clip to the tissue.

[0028] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopic endoscopic camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be located within the control tower 20. The image processing device 56 may be any computing device configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0029] The surgeon console 30 includes a first screen 32 that displays a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 provided on the robotic arm 40, and a second screen 34 that displays a user interface for controlling the surgical robotic system 10. The first screen 32 and the second screen 34 may be touch screens that allow for the display of various graphical user inputs.

[0030] The surgeon's console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of hand controls 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon's console further includes an armrest 33 for supporting the clinician's arms while operating the hand controls 38a and 38b.

[0031] The control tower 20 includes a screen 23, which may be a touchscreen and output on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40 based on a set of programmable instructions and / or input commands from the surgeon's console 30, so as to move the robotic arms 40 and corresponding surgical instruments 50 in a manner such that the robotic arms 40 and surgical instruments 50 execute a desired movement sequence in response to input from the foot pedals 36 and handle controllers 38a and 38b. The foot pedals 36 can be used to enable and lock the handle controllers 38a and 38b, reposition camera movement, and activate / deactivate electrosurgery. Specifically, the foot pedals 36 can be used to perform a clutching action on the handle controllers 38a and 38b. By depressing one of the foot pedals 36, the clutch is activated, which disconnects the handle controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the handle controllers 38a and 38b without moving the robotic arm(s) 40, and the instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0032] Each of the control tower 20, surgeon console 30 and robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network", whether plural or singular, means a data network, including but not limited to the Internet, an intranet, a wide area network, or a local area network, and is not limited to the full scope of the definition of a communication network covered by the present disclosure. Suitable protocols include, but are not limited to, the Transmission Control Protocol / Internet Protocol (TCP / IP), the Datagram Protocol / Internet Protocol (UDP / IP), and / or the Datagram Congestion Control Protocol (DC). Wireless communication can be achieved through one or more wireless configurations, for example, radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data from fixed and mobile devices over short distances using short-length radio waves, creating a personal area network (PAN)), (A set of specifications for advanced communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 Wireless Personal Area Network (WPAN) standard).

[0033] Computer 21,31,41 can include any suitable processor (not shown), the processor is operably connected to memory (not shown), the memory can include one or more of volatile, nonvolatile, magnetic, optical or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), nonvolatile RAM (NVRAM) or flash memory. The processor can be any suitable processor (e.g., control circuit) suitable for performing the operations, calculations, and / or instruction sets described in the present disclosure, including but not limited to hardware processors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), microprocessors and combinations thereof. It will be understood by those skilled in the art that a processor can be replaced by using any logical processor (e.g., control circuit) suitable for performing the algorithms, calculations, and / or instruction sets described herein.

[0034] refer to Figure 2 Each robotic arm 40 may include a plurality of links 42a, 42b, 42c interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used as known to those skilled in the art. Joint 44a is configured to secure the robotic arm 40 to the mobile cart 60 and defines a first longitudinal axis. Figure 3 , the mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 also includes a screen 69 for displaying information about the robotic arm 40. In embodiments, the robotic arm 40 may include any type and / or any number of joints.

[0035] The installation arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide lateral maneuverability of the robot arm 40. The links 62a, 62b, and 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, and 62c can move in their corresponding lateral planes parallel to each other, thereby allowing the robot arm 40 to extend relative to the patient (e.g., an operating table). In an embodiment, the robot arm 40 can be coupled to an operating table (not shown). The installation arm 61 includes a control device 65 for adjusting the movement of the links 62a, 62b, and 62c and the elevator 67. In an embodiment, the installation arm 61 may include any type and / or any number of joints.

[0036] The third link 62c can include a rotatable base 64 having two degrees of freedom. Specifically, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate about a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b can rotate about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for full three-dimensional orientation of the robotic arm 40.

[0037] Actuator 48b of joint 44b is coupled to joint 44c via strap 45a, and joint 44c is in turn coupled to joint 46b via strap 45b. Joint 44c may include a transfer case coupling straps 45a and 45b such that actuator 48b is configured to rotate each of links 42b, 42c and retainer 46 relative to one another. More specifically, links 42b, 42c, and retainer 46 are passively coupled to actuator 48b, which forces rotation about a pivot point "P" located at the intersection of a first axis defined by link 42a and a second axis defined by retainer 46. In other words, pivot point "P" is the remote center of motion (RCM) of robotic arm 40. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing surgical instrument 50 to be oriented. Due to the interconnection of the links 42a, 42b, 42c and the retainer 46 via the straps 45a and 45b, the angle between the links 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. In embodiments, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0038] The joints 44a and 44b include actuators 48a and 48b that are configured to drive the joints 44a, 44b, 44c relative to each other via a series of belts 45a and 45b or other mechanical linkages, such as drive rods, cables, or levers, etc. In particular, the actuator 48a is configured to rotate the robotic arm 40 about the longitudinal axis defined by the link 42a.

[0039] refer to Figure 2 , the holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1). The IDU 52 is configured to be coupled to the actuation mechanisms of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transmits the actuation force from its actuator to the surgical instrument 50 to actuate components of the end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a that is configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the link 42c. During an endoscopic procedure, the instrument 50 can be passed through an endoscope access port 55 ( Figure 3 ) is inserted. The holder 46 also includes a port lock 46c ( Figure 2 ).

[0040] The robot 40 also includes a mounting arm 61 and a plurality of manual override buttons 53 ( Figure 1 ), the mounting arm can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the button 53.

[0041] refer to Figure 4 Each of the computers 21, 31, and 41 of the surgical robotic system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety viewer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 regarding the current position and / or orientation of the handle controllers 38a and 38b, as well as the status of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback through the handle controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and exiting the controller 21a and, if an error in the data transmission is detected, notifies the system fault handler to place the computer 21 and / or surgical robotic system 10 into a safe state.

[0042] The controller 21a is coupled to a storage device 22a, which may be a non-transitory computer-readable medium configured to store any suitable computer data, such as software instructions executable by the controller 21a. The controller 21a also includes a transient memory 22b for loading instructions and other computer-readable data during execution of the instructions. In an embodiment, other controllers of the system 10 include similar configurations.

[0043] Computer 41 includes multiple controllers: a cart master controller 41a, a placement arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart master controller 41a receives and processes joint commands from controller 21a of computer 21 and transmits them to the placement arm controller 41b, the robotic arm controller 41c, and the IDU controller 41d. The cart master controller 41a also manages instrument changes and the overall status of the mobile cart 60, robotic arm 40, and IDU 52. The cart master controller 41a also transmits actual joint angles back to controller 21a.

[0044] Each of the joints 63a and 63b, as well as the rotatable base 64 of the mounting arm 61, is a passive joint (i.e., lacking an actuator) that allows manual user adjustment. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the mounting arm 61. When the brakes are engaged, the mounting arm controller 41b monitors the sliding of each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61. When the brakes are disengaged, the mounting arm controller 41b can be freely moved by the operator without affecting the control of other joints. The robot arm controller 41c controls each of the joints 44a and 44b of the robot arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates movement commands based on the calculated torques. These calculated motor commands are then transmitted to one or more of the actuators 48a and 48b in the robot arm 40. The actual joint positions are then transmitted by actuators 48a and 48b back to the robot controller 41c.

[0045] The IDU controller 41d receives the desired joint angles of the surgical instrument 50 (such as the wrist angle and the jaw angle) and calculates the desired current for the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart main controller 41a.

[0046] The robotic arm 40 is controlled in response to the posture of a handle controller (e.g., handle controller 38a) controlling the robotic arm 40, which is transformed into a desired posture of the robotic arm 40 by a hand-eye transformation function executed by the controller 21a. The hand-eye function, as well as other functions described herein, are implemented in software executable by the controller 21a or any other suitable controller described herein. The posture of one of the handle controllers 38a can be implemented as a coordinate position and roll-pitch-yaw (RPY) orientation relative to a coordinate reference system fixed to the surgeon's console 30. The desired posture of the instrument 50 is relative to the fixed system on the robotic arm 40. The posture of the handle controller 38a is then scaled by a zoom function executed by the controller 21a. In an embodiment, the zoom function can reduce the coordinate position and enlarge the orientation. In addition, the controller 21a can also perform a clutch function for disengaging the handle controller 38a from the robotic arm 40. In particular, if certain movement limits or other boundaries are exceeded, the controller 21a stops transmitting movement commands from the handle controller 38a to the robotic arm 40 and essentially acts like a virtual clutch mechanism, eg, limiting mechanical input from affecting mechanical output.

[0047] The desired pose of the manipulator 40 is based on the pose of the handle controller 38a and then passed through the inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the manipulator 40 to achieve the scaled and adjusted pose input by the handle controller 38a. The calculated angles are then passed to the manipulator controller 41c, which includes a joint axis controller with a proportional-derivative (PD) controller, a friction estimator module, a gravity compensator module, and a double-sided saturation block configured to limit the command torque of the motors of the joints 44a, 44b, 44c.

[0048] refer to Figure 5 , the surgical robotic system 10 is set up around an operating table 90. The system 10 includes mobile carts 60a-d, which may be numbered "1" through "4." During set up, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depends on a number of factors, such as the placement of the plurality of access ports 55a-d, which in turn depends on the procedure being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient's body, and the carts 60a-d are positioned to insert the instruments 50 and laparoscopic camera 51 into the corresponding ports 55a-d.

[0049] During use, each of the robotic arms 40a-d engages the latch 46c ( Figure 2 ) is attached to access port 55 ( Figure 3) and is attached to one of the access ports 55a-d inserted into the patient's body. The IDU 52 is attached to the holder 46, and the SIM 43 is then attached to the distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts that are configured to transmit the rotation of each motor of the IDU 52 to the instrument 50, thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40 (including the IDU 52). The SIM 43 is also configured to secure a sterile drape (not shown) to the IDU 52.

[0050] During installation, an assistant access port 50e is also inserted into the patient and is used to insert any suitable surgical instruments 70. The instruments 70 may be manual, robotic, or motor-driven surgical instruments, additional endoscopic cameras, ultrasound probes, or any other surgical instruments not directly controlled by the robotic system 10. In embodiments, more than one assistant access port 55e may be used to accommodate additional instruments.

[0051] The system 10 includes an external visual tracking unit 80 (e.g., one or more cameras or infrared sensors) configured to track the position of the access ports 50a-e, the robotic arm 40, etc. The tracking unit 80 may include one or more position sensors, which may be any suitable white light or infrared camera, electromagnetic sensor, magnetoresistive sensor, radio frequency sensor, or any other sensor suitable for adequately sensing the position of the navigation marker. The tracking unit 80 may be configured to detect markers disposed on the access ports 50a-e. The markers may be passive tracking elements (e.g., reflectors) for transmitting light signals (e.g., reflecting light emitted from the tracking unit 80). Alternatively, the markers may include radiopaque materials that are recognized and trackable by the tracking unit 80. In other configurations, active tracking markers may be employed. Active tracking markers may be, for example, light emitting diodes that emit light (e.g., infrared light). Both active and passive arrangements are possible. The markers may be arranged in a defined or known position and orientation relative to other markers to allow the tracking unit 80 to determine the position of the access ports 50a-e relative to each other. Access ports 50a-d can be aligned with a specific robotic arm 40 and corresponding instrument 50, and access port 50e is aligned with an assistant instrument 70 to allow the surgical robotic system 10 to determine the position and / or orientation of the instrument 50, camera 51, and instrument 70 within a confined space such as a surgical field.

[0052] refer to Figure 6, the first screen 32 of the surgeon's console 30 includes a GUI 100 that provides a video feed 102 of the camera 51. The video feed 102 is within the field of view of the camera 51 and may show the surgical site, instruments 50, instruments 70, etc. The video processing device 56 is configured to output the GUI 100, which may be displayed on any screen of the system 10, i.e., the first screen 32 of the surgeon's console 30, the screen 23 of the control tower 20.

[0053] Figure 7 A method for providing instructions to a bedside assistant via a GUI 100 displayed on the screen 23 of the control tower 20 is shown. The method can be implemented as software instructions executable by a processor (e.g., the controller 21a), and specifically as a software application having a GUI 100 for providing virtual instrument positions. The virtual instrument can be moved by the surgeon via the surgeon console 30 and used by the bedside assistant as a guide for moving the assistant instrument 70.

[0054] At step 200, the external positions of the access ports 50a-e are determined using the tracking unit 80 and provided to the system 10. The position data is provided to the video processing device 56 so that the virtual appliance 170 is output as a video feed 102 in the GUI 100 ( Figure 6 ) in an overlay. In particular, the access ports 50a-e are registered in a world coordinate system that allows the camera 51 and the instrument(s) 50 to be registered relative to each other.

[0055] At step 202, the controller 21a determines whether the instrument 70 is within the field of view of the camera 51. The video processing device 56 may use a machine learning image processing algorithm. In an embodiment, the machine learning may include a convolutional neural network (CNN) and / or a support vector machine (SVM). The CNN may be trained on previous data (e.g., images of various instruments and devices). The video processing device 56 is configured to communicate with the controller 21a to notify the controller 21a whether the instrument 70 is present.

[0056] If the instrument 70 is outside the field of view of the camera 51, the controller 21a may output a message to the GUI 100 at the screen 23 of the control tower 20 and / or the screen 32 of the surgeon's console 30 at step 204. The message may include a directional arrow for moving the camera 51 and / or the instrument 70 so that the instrument 70 is within the field of view of the camera 51.

[0057] Once the instrument 70 is visible to the camera 51, at step 206, the video processing device 56 renders the virtual instrument 170 in the video feed 102 to generate an augmented reality video feed. The video processing device 56 is configured to load the 3D model and physical model and kinematics of the assistant's instrument 70, which are used to render the virtual instrument 170 and simulate the movement of the virtual instrument 170. The virtual instrument 170 can be rendered in a virtual 3D space within the field of view of the camera 51 based on a depth map of the surgical site. Image processing techniques can be used for stereoscopic video feeds to perform depth mapping. As described above, the virtual instrument 170 is simultaneously displayed on the first screen 32 of the surgeon's console 30 and the screen 23 of the control tower 20, each of which receives a feed from the camera 51.

[0058] At step 208, the surgeon can control the virtual instrument 170. This can be done in the same manner as controlling the robotic arm 40 and the instrument 50 and / or camera 51 coupled thereto. The surgeon selects the virtual instrument 170 via one of the handle controllers 38a or 38b, which allows the virtual instrument 170 to move as if it were an actual instrument, such as the instrument 50. The movement of the handle controller 38a or 38b causes the virtual instrument 170 to move in the augmented video feed 102 based on the depth map of the scene and the 3D rendering of the model.

[0059] The video processing device 56 is configured to render a physics-based virtual 3D space based on a stereoscopic reconstruction (i.e., a depth map) from the video feed 102. This will enable the creation of a virtual boundary in the virtual 3D space that the virtual instrument 170 cannot move through. The virtual boundary can be indicated to the user via tactile feedback through the handle controllers 38a and 38b and by adjusting the resistance of the motors of the controllers 38a and 38b. Thus, the surgeon console 30 can limit the movement of the virtual instrument 170 based on the virtual boundary. In an embodiment, the virtual instrument 170 can be used as a virtual tool to measure distances in the video feed 102. This can be achieved through any number of 3D user interface methods, such as moving the virtual instrument 170 between two points to measure the distance like a virtual ruler.

[0060] The movement of the virtual instrument 170 may also include multiple movements or manipulations in one or more directions until the virtual instrument 170 is in the desired position. At step 210, the final position of the virtual instrument 170 is displayed on the GUI 100, and in particular, on the video feed 102. In addition, the sequence of movements taken to reach the final position may also be replayed on the GUI 100. These aids provide guidance to the assistant who is moving the actual (i.e., tangible) assistant instrument 70 to the final position indicated by the virtual instrument 170. The GUI 100 may display directional arrows that indicate which way the instrument 70 will be moved to reach the final position.

[0061] At step 212, the video processing device 56 checks whether the assistant instrument 70 is in the same or substantially the same position as the virtual instrument 170. The video processing device 56 compares a plurality of key points of the assistant instrument 70 with corresponding key points of the virtual instrument 170 to determine whether there is a 3D alignment between the instrument 70 and the virtual instrument 170. At step 214, if there is a mismatch between the position of the instrument 70 and the virtual instrument 170, the GUI 100 may indicate this using an alert or by using a particular color (e.g., red) of the virtual instrument 170 on the video feed 102. If there is alignment, at step 216, the GUI 100 may indicate this by changing the color (e.g., green), by a message, or the like. In an embodiment, the color of the virtual instrument 170 may be similar to a thermal image. Figure 1 The color changes from red to green to indicate the closeness of alignment.

[0062] In an embodiment, in addition to or as an alternative to the video processing device 56 checking whether the assistant instrument 70 is in the same or substantially the same position as the virtual instrument 170, the system 10 may display a prompt to the user on the surgeon's console 30 to confirm whether the assistant instrument 70 is in the indicated position of the virtual instrument 170. The prompt may be generated periodically and / or in response to the assistant instrument 70 approaching the position of the virtual instrument 170. Once the user affirmatively confirms that the assistant instrument 70 is in the indicated position, the GUI 100 may indicate this by changing color (e.g., green) or another message. In an embodiment, the color of the virtual instrument 170 may be similar to a thermal image. Figure 1 The color changes from red to green to indicate the closeness of alignment.

[0063] The disclosed system and method can be used with a variety of different surgical instruments, and several exemplary embodiments are provided below. Endoscopic stapling involves a complex workflow that includes positioning the stapling end effector at a precise angle and position. Using a virtual stapler will allow an assistant to follow the assistant instrument 70 (e.g., a physical stapler) to the exact location of the tissue that needs to be stapled.

[0064] In another embodiment, the virtual instrument 170 can be a virtual laparoscopic ultrasound probe, and the instrument 70 can be an endoscopic ultrasound probe. A robotic system can use a robotic insertion probe that is manipulated by a grasping instrument to obtain an ultrasound image of the tissue. However, such probes are expensive and inconvenient to use, and they also require one of the robotic arms. Therefore, the assistant instrument 70 can be an endoscopic ultrasound probe, and the assistant can position the endoscopic ultrasound probe as indicated by the virtual instrument 170, thereby indicating the location where ultrasound imaging is required. This will allow the surgeon to move the virtual instrument 170 (e.g., an ultrasound probe) along the organ to track or confirm the tumor boundary or other critical structures.

[0065] In additional embodiments, the virtual instrument 170 can be a virtual endoscope, and the assistant instrument 70 can be an additional endoscope that can be controlled by the assistant or can be controlled directly by the surgeon through the surgeon's console 30. The surgeon can disconnect the clutch and manipulate the virtual endoscope without the involvement of the assistant. The virtual endoscope can be used to render views of the surgical site from different angles without moving the camera 51. In embodiments, the feed from the virtual endoscope can be combined with the feed from the camera 51 to generate a surround view of the surgical site or simulate port hopping, i.e., switching the video feed displayed on the first screen 32 of the surgeon's console 30.

[0066] The virtual endoscope can also be used to render realistic views of anatomical structures from multiple angles using continuously updated depth maps. Machine learning can be implemented in the virtual endoscope, such as methods based on generative adversarial networks (GANs), to render views that are not directly visible by the camera 51. The GAN can be trained on the anatomical structures of other patients from previous videos of similar surgeries to generate such views.

[0067] It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be interpreted as limiting, but merely as illustrations of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A surgical robot system, comprising: an assistant access port configured to receive an assistant instrument; an endoscopic camera configured to generate a video feed of the surgical site and the assistant instrument; a control tower, the control tower including a first screen; a surgeon console comprising a second screen and a handle controller, the surgeon console configured to receive user input; and A video processing device, the video processing device being configured to: rendering a virtual instrument of the assistant instrument in the video feed to generate an augmented video feed; moving the virtual instrument in the augmented video feed in response to the user input; outputting the augmented video feed with the virtual instrument on the first screen and the second screen; confirming whether the assistant instrument is placed at the position of the virtual instrument; and Whether the assistant instrument is placed at the position of the virtual instrument is indicated on the first screen and the second screen.

2. The surgical robot system according to claim 1, further comprising: a robotic arm comprising a robotic instrument; as well as A robotic access port is configured to receive the robotic instrument.

3. The surgical robot system according to claim 2, wherein: The surgeon console is configured to switch between controlling the robotic instrument and the virtual instrument.

4. The surgical robot system according to claim 2, further comprising a tracking unit configured to track the position of the assistant access port and the position of the robot access port.

5. The surgical robot system according to claim 4, wherein: The video processing device is configured to determine the positioning of the assistant instrument based on the position of the assistant access port and the position of the robot access port.

6. The surgical robot system according to claim 1, wherein: The video processing device is configured to render the virtual instrument based on the 3D model data of the assistant instrument.

7. The surgical robot system according to claim 1, wherein: The endoscopic camera is a stereo camera, and the processing device is configured to generate a depth map of the surgical site.

8. The surgical robot system according to claim 7, wherein: The processing device is configured to generate a virtual boundary corresponding to a physical boundary of the surgical site based on the depth map.

9. The surgical robot system according to claim 8, wherein: The surgeon console is configured to limit the user input for controlling movement of the virtual instrument at the surgeon console based on the virtual boundary.

10. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a computer-implemented method for transmitting movement instructions using a virtual implement, the method comprising: Receive video feed of the surgical site and assistant instruments; rendering a virtual instrument of the assistant's instrument in the video feed to generate an augmented video feed that is displayed on a first screen of the control tower and a second screen of the surgeon's console; moving a rendered virtual instrument in the video feed in response to input signals received from the surgeon console; outputting the augmented video feed with the virtual instrument on the first screen and the second screen; Prompting the user to confirm whether the assistant instrument is placed at the position of the virtual instrument; and Whether the assistant instrument is placed at the position of the virtual instrument is indicated on the first screen and the second screen.

11. The non-transitory computer readable medium of claim 10, wherein: The computer-implemented method further comprises: Switching between controlling a robotic instrument coupled to the robotic arm and inserted through the robotic access port and the virtual instrument.

12. The non-transitory computer readable medium of claim 11, wherein: The computer-implemented method further comprises: The position of the assistant access port through which the assistant instrument is inserted and the position of the robot access port are tracked.

13. The non-transitory computer readable medium of claim 12, wherein: The computer-implemented method further comprises: The positioning of the assistant instrument is determined based on the position of the assistant access port and the position of the robot access port.

14. The non-transitory computer readable medium of claim 10, wherein: The computer-implemented method further comprises: The virtual instrument is rendered based on the 3D model data of the assistant instrument.

15. The non-transitory computer readable medium of claim 10, wherein: The computer-implemented method further comprises: A depth map of the surgical site is generated.

16. The non-transitory computer readable medium of claim 15, wherein: The computer-implemented method further comprises: A virtual boundary corresponding to the physical boundary of the surgical site is generated based on the depth map.

17. The non-transitory computer readable medium of claim 16, wherein: The computer-implemented method further comprises: Movement of the virtual instrument at the surgeon console is constrained based on the virtual boundary.

18. A surgical robot system comprising: a robotic arm comprising a robotic instrument; as well as a robotic access port configured to receive the robotic instrument; an assistant access port configured to receive an assistant instrument; an endoscopic camera configured to generate a video feed of the surgical site and the assistant instrument; a control tower, the control tower including a first screen; a surgeon console comprising a second screen and a handle controller, the surgeon console configured to receive user input to control the robotic instrument and the virtual instrument; as well as A video processing device, the video processing device being configured to: rendering the virtual instrument of the assistant instrument in the video feed to generate an augmented video feed; moving the virtual instrument in the augmented video feed in response to the user input; outputting the augmented video feed with the virtual instrument on the first screen and the second screen; confirming whether the assistant instrument is placed at the position of the virtual instrument; and Whether the assistant instrument is placed at the position of the virtual instrument is indicated on the first screen and the second screen.

19. The surgical robot system according to claim 18, wherein: The endoscopic camera is a stereo camera and the processing device is configured to generate a depth map of the surgical site.

20. The surgical robot system according to claim 19, wherein: The processing device is configured to generate a virtual boundary corresponding to a physical boundary of the surgical site based on the depth map, and the surgeon console is configured to limit the user input for controlling movement of the virtual instrument at the surgeon console based on the virtual boundary.

Images