Surgical robotic systems and methods for laparoscopic instrument and camera navigation

Abstract

A surgical robotic system includes a surgical robot arm having a surgical device and an instrument drive unit configured to actuate the surgical device. The system also includes a surgeon console configured to receive user input to control at least one of the surgical robot arm or the surgical device. The system further includes a control tower coupled to the surgical robot arm and the surgeon console. The control tower includes a storage device and a controller configured to store a plurality of checkpoints on the storage device. Each checkpoint of the plurality of checkpoints includes image data of a surgical site and position data corresponding to a previous position of the surgical robot during a surgical procedure. The controller is further configured to output a graphical user interface (GUI) on the display screen. The GUI is configured to display image data of at least one checkpoint of the plurality of checkpoints and receive a user input selecting the checkpoint. The controller is further configured to control the surgical robot arm to move the surgical robot arm to a previous position based on the selected checkpoint.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 517,158, filed on August 2, 2023. The entire contents of the above application are incorporated herein by reference. Background Art

[0003] Surgical robotic systems are currently being used for various surgical procedures, including minimally invasive surgery. Some surgical robotic systems include a surgeon's console that controls a surgical robotic arm and a surgical instrument having an end effector (e.g., a clamp or a grasping instrument) that is connected to the robotic arm and actuated by the robotic arm. During operation, the robotic arm is moved to a position above the patient, and then the surgical instrument is guided into a small incision via a surgical port or a natural orifice of the patient to position the end effector at the working site in the patient's body. A laparoscopic camera (which is also held by one of the robotic arms) is inserted into the patient to image the surgical site. During robotic surgery, the doctor continuously monitors the surgeon's display screen set on the surgeon's console, and gradually guides the camera to the anatomical structure of interest as the surgeon navigates between other ports and their corresponding instruments. Therefore, the instrument and camera may be moved multiple times (e.g., hundreds of times). This process makes robotic surgery complex and time-consuming. In addition, the camera may be fogged, so it must be removed from the patient's cavity to clean it and reposition it in the same position. Placing and removing the camera requires multiple moving steps, and the surgeon needs to retrace these moving steps. The multiple movements of instruments and cameras not only increase the duration of the procedure, but also increase the amount of anesthesia required for the patient, which in turn increases the recovery duration and cost of the procedure. Summary of the invention

[0004] The present disclosure provides a surgical robotic system configured to store navigation position checkpoints of a laparoscopic camera and navigation position checkpoints of one or more instruments inserted into a patient. These checkpoints can be saved manually and / or automatically. The system provides a user interface for surgeons and staff that can be used to save the positions of instruments and cameras. The user interface can also be used to retract the camera and instruments at any time during a surgical procedure so that the system automatically moves the camera and / or instruments to the selected checkpoints.

[0005] In one embodiment, the checkpoint process can be used for removal (e.g., cleaning) and reinsertion. In this embodiment, the initial position of the camera when the camera is outside the patient's cavity is recorded as the first checkpoint. Then, the robotic arm inserts the camera into the patient to perform the surgical operation. If the camera becomes blurred, the surgeon can recall the camera to its initial position (i.e., the first checkpoint). One or more subsequent checkpoints of the camera position can also be recorded after insertion to generate a so-called breadcrumb navigation path, along which the system can automatically navigate the camera in a backward or forward direction. Therefore, once the camera cleaning is completed, the surgeon can return to the latest checkpoint position so that the arm will automatically move to the desired (e.g., last known) recorded position. The same navigation path can be used for the insertion and navigation of the instrument. The term "path" means a route formed using the coordinates of an instrument or camera saved during surgery.

[0006] The checkpoint system can also be used to recover the system after a power failure or after a serious error has occurred in the system. In the event of a power failure, the emergency backup battery may not allow the surgeon to operate the arm. Checkpoints can be created for each of the robotic arms and instruments, allowing for an automated recovery process.

[0007] According to one embodiment of the present disclosure, a surgical robot system is disclosed. The surgical robot system includes a surgical robot arm having a surgical device and an instrument drive unit configured to actuate the surgical device. The system also includes a surgeon's console, which is configured to receive user input to control at least one of the surgical robot arm or the surgical device. The system further includes a control tower, which is connected to the surgical robot arm and the surgeon's console. The control tower includes a storage device and a controller, which is configured to store multiple checkpoints on the storage device. Each of the multiple checkpoints includes image data of the surgical site and position data corresponding to the previous position of the surgical robot arm during the surgical operation. The controller is also configured to output a graphical user interface (GUI) on a display screen. The GUI is configured to display image data of at least one of the multiple checkpoints and receive user input for selecting the checkpoint. The controller is further configured to control the surgical robot arm based on the selected checkpoint to move the surgical robot arm to a previous position.

[0008] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the controller may be further configured to store a checkpoint in response to activation of a surgical device. The controller may also be configured to store a checkpoint in response to a user command. The controller may be additionally configured to store a checkpoint based on the elapsed time of the surgical procedure. The controller may be further configured to detect a stage of the surgical procedure. The controller may also be configured to store a checkpoint based on the detected stage of the surgical procedure. The surgical device may be a surgical robotic instrument or a laparoscopic camera. The checkpoint may include position data corresponding to a laparoscopic camera external to the patient.

[0009] According to another embodiment of the present disclosure, a method for restoring the position state of a surgical robot system is disclosed. The method includes storing a plurality of checkpoints on a storage device, the plurality of checkpoints including image data of a surgical site and position data corresponding to a previous position of a laparoscopic camera held by a surgical robot arm during a surgical operation. A first checkpoint corresponds to a position of the laparoscopic camera outside a patient, and a second checkpoint of the plurality of checkpoints corresponds to a position of the laparoscopic camera inside a patient. The method further includes outputting a graphical user interface (GUI) configured to display the image data of the plurality of checkpoints on a display screen. The method additionally includes receiving a first user input selecting at least one of the first checkpoint or the second checkpoint through the GUI, and controlling the surgical robot arm based on the selection of at least one of the first checkpoint or the second checkpoint to move the surgical robot arm to a previous position.

[0010] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include storing the first checkpoint or the second checkpoint in response to activation of the laparoscopic camera. The method may further include storing the first checkpoint or the second checkpoint in response to a user command. The method may additionally include storing the first checkpoint or the second checkpoint based on an elapsed time of the surgical procedure. The method may also include detecting a stage of the surgical procedure; and storing the first checkpoint or the second checkpoint based on the detected stage of the surgical procedure.

[0011] According to another embodiment of the present disclosure, a method for restoring a position state of a surgical robot system is disclosed. The method includes storing a plurality of checkpoints on a storage device. Each of the plurality of checkpoints includes image data of a surgical site and position data corresponding to a previous position of the surgical robot arm during a surgical procedure. The method further includes outputting a graphical user interface (GUI) configured to display image data of at least one of the plurality of checkpoints on a display screen. The method additionally includes receiving user input selecting the at least one checkpoint through the GUI, and controlling the surgical robot arm based on the selected at least one checkpoint to move the surgical robot arm to a previous position.

[0012] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include storing a checkpoint in response to activation of a surgical device held by a surgical robot arm. The method may further include storing a checkpoint in response to a user command. The method may additionally include storing a checkpoint based on an elapsed time of a surgical procedure. The method may further include detecting a stage of a surgical procedure and storing a checkpoint based on the detected stage of the surgical procedure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0014] Figure 1 is a perspective view of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system comprising a control tower, a control console, and one or more surgical robot arms, each of the one or more surgical robot arms being disposed on a mobile cart;

[0015] Figure 2 According to the embodiments of the present disclosure Figure 1 A three-dimensional diagram of a surgical robot arm of a surgical robot system;

[0016] Figure 3 is a perspective view of a mobile cart having a mounting arm with a mounting arm according to an embodiment of the present disclosure Figure 1 The surgical robot arm of the surgical robot system;

[0017] Figure 4 According to the embodiments of the present disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system;

[0018] Figure 5 According to the embodiments of the present disclosure Figure 1 A schematic plan view of a surgical robot system, the surgical robot system being positioned around an operating table;

[0019] Figure 6 is a schematic diagram of a system for determining a stage of a surgical procedure according to an embodiment of the present disclosure;

[0020] Figure 7 is a flow chart of a method for saving a checkpoint according to an embodiment of the present disclosure;

[0021] Figure 8A-1 , Figure 8A-2 and Figure 8B each showing a flow chart of a method for restoring a checkpoint according to an embodiment of the present disclosure; and

[0022] Figures 9 to 20 According to the implementation of the embodiment of the present disclosure Figure 7 and a screenshot of a graphical user interface (GUI) for the method of FIG. 8 . DETAILED DESCRIPTION

[0023] Embodiments of the surgical robotic system disclosed herein are described in detail with reference to the drawings, in which like reference numerals designate like or corresponding elements in each of the several views.

[0024] Reference Figure 1 , the surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including a surgeon's console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 to which a surgical instrument 50 is removably coupled. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or robotic arms 40.

[0025] The surgical instrument 50 is configured to be used during minimally invasive surgery. In an embodiment, the surgical instrument 50 can be configured for open surgery. In another embodiment, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by compressing the tissue between the jaw members and applying an electrosurgical current thereto. In yet another embodiment, the surgical instrument 50 can be a surgical stapler, which includes a pair of jaws configured to grasp and clamp tissue, simultaneously deploy multiple tissue fasteners (e.g., staples) and cut stapled tissue. In yet another embodiment, the surgical instrument 50 can be a surgical clamp applicator, which includes a pair of jaws configured to apply a surgical clamp to the tissue.

[0026] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture a video of the surgical site. The laparoscopic camera 51 may be a stereo camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be disposed within the control tower 20. The image processing device 56 may be any computing device configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0027] The surgeon console 30 includes a first screen 32 that displays a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 disposed on the robotic arm 40, and a second screen 34 that displays a user interface for controlling the surgical robotic system 10. The first screen 32 and the second screen 34 may be touch screens that allow various graphical user inputs to be displayed.

[0028] The surgeon console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of hand controls 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon console further includes an armrest 33 for supporting the clinician's arms while operating the hand controls 38a and 38b.

[0029] The control tower 20 includes a screen 23 (which may be a touch screen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as moving the robotic arms 40 and corresponding surgical instruments 50 based on a set of programmable instructions and / or input commands from the surgeon console 30, in a manner such that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from the foot pedals 36 and hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, reposition camera movement, and activation / deactivation of electrosurgery. In particular, the foot pedals 36 may be used to perform a clutch action on the hand controllers 38a and 38b. The clutch is initiated by pressing one of the foot pedals 36, which disconnects the hand controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0030] Each of the control tower 20, surgeon console 30 and robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network", whether plural or singular, means a data network, including but not limited to the Internet, an intranet, a wide area network or a local area network, and is not limited to the full scope of the definition of communication networks as covered by the present disclosure. Suitable protocols include but are not limited to the Transmission Control Protocol / Internet Protocol (TCP / IP), the Datagram Protocol / Internet Protocol (UDP / IP) and / or the Datagram Congestion Control Protocol (DC). Wireless communication can be achieved through one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data from fixed and mobile devices over short distances using short-length radio waves, creating a personal area network (PAN)), (A set of specifications for advanced communications protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 Wireless Personal Area Network (WPAN) standard).

[0031] Computer 21,31,41 may include any suitable processor (not shown), which is operably connected to a memory (not shown), which may include one or more of a volatile, nonvolatile, magnetic, optical or electrical medium, such as a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM (EEPROM), a nonvolatile RAM (NVRAM) or a flash memory. The processor may be any suitable processor (e.g., a control circuit) suitable for performing the operations, calculations and / or instruction sets described in the present disclosure, including but not limited to a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor and a combination thereof. It will be appreciated by those skilled in the art that a processor may be replaced by using any logical processor (e.g., a control circuit) suitable for performing the algorithms, calculations and / or instruction sets described herein.

[0032] Reference Figure 2 Each robot arm 40 may include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used as known to those skilled in the art. Joint 44a is configured to secure the robot arm 40 to the mobile cart 60 and defines a first longitudinal axis. Figure 3, the mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 also includes a screen 69 that is used to display information about the robotic arm 40. In an embodiment, the robotic arm 40 may include any type and / or number of joints.

[0033] The installation arm 61 includes a first link 62a, a second link 62b and a third link 62c, which provide lateral maneuverability of the robot arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c can move in their corresponding lateral planes parallel to each other, thereby allowing the robot arm 40 to extend relative to the patient (e.g., an operating table). In an embodiment, the robot arm 40 can be connected to an operating table (not shown). The installation arm 61 includes a control device 65, which is used to adjust the movement of the links 62a, 62b, 62c and the elevator 67. In an embodiment, the installation arm 61 can include any type and / or number of joints.

[0034] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate around a first fixed arm axis perpendicular to a plane defined by the third link 62c, while the second actuator 64b can rotate around a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow the robot 40 to achieve full three-dimensional orientation.

[0035] The actuator 48b of the joint 44b is connected to the joint 44c via the belt 45a, and the joint 44c is in turn connected to the joint 46b via the belt 45b. The joint 44c can include a transfer case connecting the belts 45a and 45b, so that the actuator 48b is configured to rotate each of the links 42b, 42c and the holder 46 relative to each other. More specifically, the links 42b, 42c and the holder 46 are passively connected to the actuator 48b, which is forced to rotate about the pivot point "P", which is located at the intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point "P" is the remote center of motion (RCM) of the robot 40. Thus, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing the surgical instrument 50 to be oriented. Due to the interconnection of the links 42a, 42b, 42c and the retainer 46 via the straps 45a and 45b, the angle between the links 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. In an embodiment, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0036] The joints 44a and 44b include actuators 48a and 48b that are configured to drive the joints 44a, 44b, 44c relative to each other via a series of belts 45a and 45b or other mechanical linkages such as drive rods, cables or levers. In particular, the actuator 48a is configured to rotate the robot arm 40 about a longitudinal axis defined by the link 42a.

[0037] Reference Figure 2 The holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1 ). The IDU 52 is configured to be coupled to an actuation mechanism of the surgical instrument 50 and the camera 51, and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transmits the actuation force from its actuator to the surgical instrument 50 to actuate a component of the surgical instrument 50 (the end effector 49). The holder 46 includes a sliding mechanism 46a configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the connecting rod 42c. During laparoscopic surgery, the instrument 50 can be inserted through a laparoscopic access port 55 ( Figure 3 ) is inserted. The holder 46 also includes a port latch 46c ( Figure 2 ).

[0038] The robotic arm 40 also includes a plurality of manual override buttons 53 ( Figure 1), these manual override buttons are provided on the IDU 52 and the mounting arm 61 and can be used in a manual mode. A user can press one or more of these buttons 53 to move the component associated with the button 53.

[0039] Reference Figure 4 , each of the computers 21, 31, 41 of the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety viewer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 about the current position and / or orientation of the hand controllers 38a and 38b and the state of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the required drive commands for each joint and / or IDU 52 of the robotic arm 40, and transmits these commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback through the hand controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and exiting the controller 21a and, if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or surgical robot system 10 into a safe state.

[0040] The controller 21a is coupled to a memory 22a, which may be a non-transitory computer readable medium configured to store any suitable computer data, such as software instructions executable by the controller 21a. The controller 21a also includes a transient memory 22b for loading instructions and other computer readable data during execution of the instructions. In an embodiment, other controllers of the system 10 include similar configurations.

[0041] The computer 41 includes a plurality of controllers, namely, a cart master controller 41a, a placement arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart master controller 41a receives and processes joint commands from the controller 21a of the computer 21, and transmits them to the placement arm controller 41b, the robot arm controller 41c, and the IDU controller 41d. The master cart controller 41a also manages instrument exchange and moves the overall status of the cart 60, the robot arm 40, and the IDU 52. The cart master controller 41a also transmits the actual joint angles back to the controller 21a.

[0042] Each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61 is a passive joint (i.e., there is no actuator therein), thereby allowing the user to manually adjust it. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the mounting arm 61. The mounting arm controller 41b monitors the sliding of each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61 when the brake is engaged, or can be freely moved by the operator when the brake is disengaged, but does not affect the control of other joints. The robot arm controller 41c controls each joint 44a and 44b of the robot arm 40 and calculates the required motor torque required for gravity compensation, friction compensation and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates a movement command based on the calculated torque. The calculated motor command is then transmitted to one or more of the actuators 48a and 48b in the robot arm 40. The actual joint positions are then transmitted by actuators 48a and 48b back to the robot controller 41c.

[0043] The IDU controller 41d receives the required joint angles (such as wrist angle and jaw angle) of the surgical instrument 50 and calculates the required current for the motors in the IDU 52. The IDU controller 41d calculates the actual angle based on the motor position and transmits the actual angle back to the cart main controller 41a.

[0044] Reference Figure 5 , the surgical robotic system 10 is set up around an operating table 90. The system 10 includes mobile carts 60a to 60d, which may be numbered "1" to "4". During set up, each of the carts 60a to 60d is positioned around the operating table 90. The position and orientation of the carts 60a to 60d depends on a number of factors, such as the placement of a plurality of access ports 55a to 55d, which in turn depends on the surgery being performed. Once the port placement is determined, the access ports 55a to 55d are inserted into the patient and the carts 60a to 60d are positioned to insert the instruments 50 and the laparoscopic camera 51 into the corresponding ports 55a to 55d.

[0045] During use, each of the robotic arms 40a to 40d is attached to one of the access ports 55a to 55d by latching 46c ( Figure 2 ) is attached to the entry port 55 ( Figure 3) and insert the access port into the patient. The IDU 52 is attached to the holder 46, and the SIM 43 is then attached to the distal end of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts configured to transmit the rotation of the individual motors of the IDU 52 to the instrument 50, thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40 (including the IDU 52). The SIM 43 is also configured to secure a sterile sheet (not shown) to the IDU 52.

[0046] A surgical procedure may include multiple stages, and each stage may include one or more surgical actions. As used herein, the term "stage" refers to a surgical event consisting of a series of steps (e.g., closure). A "surgical action" may include incision, compression, anastomosis, clamping, suturing, cauterization, sealing, or any other such action performed to complete a stage in a surgical procedure. A "step" refers to completing a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 50 (e.g., clamps) are used to achieve a specific goal by performing one or more surgical actions.

[0047] Reference Figure 6 , the surgical robot system 10 may include a machine learning (ML) processing system 310 that processes surgical data using one or more ML models to identify one or more features in the surgical data, such as surgical stages, instruments, anatomical structures, etc. The ML processing system 310 includes an ML training system 325, which may be a separate device (e.g., a server) that stores its output as one or more trained ML models 330. The ML models 330 may be accessed by an ML execution system 340. The ML execution system 340 may be separate from the ML training system 325, i.e., the device that "trains" the model is separate from the device that performs "inference", i.e., performs real-time processing of surgical data using the trained ML models 330.

[0048] System 10 includes a data receiving system 305 that collects surgical data, including video data and surgical instrument data. Data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located in and / or associated with a surgical operating room and / or a control center. Data receiving system 305 may receive surgical data in real time, i.e., receive surgical data while a surgical procedure is being performed.

[0049] In some examples, the ML processing system 310 may further include a data generator 315 for generating simulated surgery data (e.g., a set of virtual images) or recording video data from the video processing device 56 and other data sources (e.g., user input, arm movement, etc.) to train the ML model 330. The data generator 315 may access (read / write) the data storage 320 to record data, including multiple images and / or multiple videos.

[0050] The ML processing system 310 also includes a phase detector 350 that uses the ML model to identify phases within a surgical procedure. The phase detector 350 uses a specific surgical tracking data structure 355 from the list of surgical tracking data structures. The phase detector 350 selects the surgical tracking data structure 355 based on the type of surgical procedure being performed. In one or more examples, the type of surgical procedure is predetermined or input by a user. The surgical tracking data structure 355 identifies a set of potential phases that may correspond to a portion of a particular type of surgical procedure.

[0051] In some examples, the surgical tracking data structure 355 can be a graph including a set of nodes and a set of edges, where each node corresponds to a potential stage. These edges can provide directional connections between nodes, which indicate (via directions) the expected order of the stages to be encountered throughout the surgical iteration. The surgical tracking data structure 355 may include one or more branch nodes that feed into multiple next nodes and / or may include one or more divergence points and / or convergence points between nodes. In some instances, the stage indicates a surgical action (e.g., surgical action) being performed or already performed and / or indicates a combination of actions that have been performed. In some examples, the stage is related to the biological state of the patient undergoing surgery. For example, the biological state can indicate complications (e.g., blood clots, arterial / venous obstruction, etc.), prerequisites (e.g., lesions, polyps, etc.). In some examples, the ML model 330 is trained to detect "abnormal conditions" such as bleeding, arrhythmias, vascular abnormalities, etc.

[0052] The phase detector 350 outputs a phase prediction associated with the portion of the video data analyzed by the ML processing system 310. The phase prediction is associated with the portion of the video data by identifying the start time and the end time of the portion of the video analyzed by the ML execution system 340. The output phase prediction may include an identification of a surgical phase detected by the phase detector 350 based on the output of the ML execution system 340. Further, in one or more examples, the phase prediction may include an identification of a structure (e.g., an instrument, an anatomical structure, etc.) identified by the ML execution system 340 in the analyzed portion of the video. The phase prediction may also include a confidence score for the prediction. Other examples may include various other types of information in the output phase prediction. The controller 21a may use the predicted phase to determine when to implement the collision detection method described below.

[0053] Figure 7 A method for saving an inspection point of a robotic arm 40 controlling a camera 51 is shown. The method may be implemented as software instructions executable by the controller 21a or any other suitable processor of the system 10. Although the method is described with respect to moving the camera 51 to a saved inspection point and its corresponding robotic arm 40, the method may also be extended to controlling other robotic arms 40 and their instruments 50.

[0054] Reference Figure 7 , the method is implemented to be performed during operation of the system 10 (i.e., during a surgical procedure). At step 100, the controller 21a determines that the surgeon is operating the robotic arm 40, i.e., by detecting movement, sliding, or button pressing at one or both of the handle controllers 38a and 38b. At step 102, the controller 21a attempts to obtain a surgical stage name from the stage detector 350. At step 104, if no information is received (e.g., if the stage detector 350 does not exist or there is no corresponding available stage), the controller 21a proceeds to step 106 to enable manual naming of the stage. The surgeon or any other user can identify and name a checkpoint group by entering (e.g., voice input, typing, etc.) a stage name or another identifier. The checkpoint group organizes the one or more checkpoints based on their stage.

[0055] If the phase detector 350 is active and can identify the current phase, the controller 21a uses the current phase to create a checkpoint group at step 108. At step 110, checkpoints are generated and saved in the memory 22a as part of the checkpoint group. Each checkpoint includes data describing the orientation and position of the robotic arm 40, such as coordinates in a world coordinate system, joint angles, arm height, angle relative to the table 90, etc. The checkpoint also includes position information for the IDU 52, such as the angular motor position of one or more motors controlling the camera 51 and / or the instrument 50. The checkpoint also includes image data for the checkpoint, such as video and / or still images taken during the checkpoint creation, such as screenshots (e.g., thumbnails) from the video. At step 112, the checkpoints can also be saved as part of a new checkpoint group, i.e., checkpoints that occur during a particular phase of the surgery.

[0056] Checkpoints may be automatically generated and stored based on detection of a particular phase as at step 114. In particular, at step 116, a checkpoint is generated in response to the insertion or removal of the camera 51 and instrument 50 into or out of the access port 50a to 50d. Another automatic checkpoint generation routine may be performed at step 118, which includes creating a checkpoint after a set time period has passed, which may be performed periodically (e.g., every minute). In addition to automatic checkpoint generation, the user may also generate a checkpoint by commanding the surgeon console 30 via Figures 9 to 20 The GUI 200 of FIG. 1 generates the checkpoint while the checkpoint is manually generated at step 120. Specifically, Fig.12 GUI 200 is shown including buttons 203a and 203b for generating a manual checkpoint and manually saving a checkpoint, respectively. The system 10 may also continuously record and store checkpoints in a dynamic, continuous manner, such that a user may choose to configure or restore the system 10 to any previous point in a surgical procedure.

[0057] Figure 8A-1 , Figure 8A-2 and Figure 8B A method for restoring an inspection point of a robotic arm 40 controlling a camera 51 is shown. The method may be implemented as software instructions executable by the controller 21a or any other suitable processor of the system 10. Although the method is described with respect to restoring a camera 51 and its corresponding robotic arm 40 to a selected inspection point, the method may also be extended to controlling other robotic arms 40 and their instruments 50.

[0058] The surgeon or staff can use Figures 9 to 20The user can browse and restore any desired checkpoints by using the GUI 200 displayed as an overlay on the screen 32 of the surgeon console 30 or any other suitable screen of the system 10. The screen 32 can overlay the GUI 200 on the video feed from the camera 51, or as a separate menu screen. At step 130, the user activates the GUI 200 by pressing the checkpoint button 202 via any suitable input method (e.g., touch input, pointer, etc.).

[0059] At step 132, upon being pressed, the GUI 200 displays various types of previously saved checkpoints. Exemplary categories include, but are not limited to, periodic checkpoints saved during step 118, manual checkpoints saved during step 120, instrument event checkpoints saved during step 116, session checkpoints saved during step 114, and dynamic checkpoints. Each category may be further accessed via its respective buttons 202a through 202e, respectively.

[0060] At step 134, the controller 21a receives input selecting one of the buttons 202a to 202d. If the selected input is not a dynamic checkpoint option via button 202e, the controller 21a checks whether the selected input is a stage checkpoint selected via button 202d. If so, the controller proceeds to step 138 during which the saved stage checkpoint group 204a to 204f is displayed (e.g., overlaid) on the GUI 200, as shown. Fig.15 Each of these groups 204a to 204f may be represented by an image (i.e., a thumbnail screenshot) of the video corresponding to the checkpoint group and lists the affected instruments 50 and / or cameras 51. At step 140, the user selects which of the checkpoint groups 204a to 204f is to be used to display the checkpoints captured during the phase. Each of these checkpoint groups may include one or more checkpoints.

[0061] Returning to step 136, if the selected button is one of buttons 202a to 202c (selecting a periodic checkpoint, a manual checkpoint, or an instrument checkpoint, respectively), the controller proceeds to step 142 during which the user selects one or more associated checkpoints, such as Fig.11 , 13 and 14.

[0062] Fig.11Periodic checkpoints 205a-205c are shown, displayed as a submenu of the periodic checkpoint button 202a. Each checkpoint 205a-205c may be represented by an image (i.e., a thumbnail screenshot) of the video corresponding to the checkpoint and lists the affected instruments 50 and / or cameras 51. The GUI 200 may include arrows 205d and 205e for scrolling through additional checkpoints. Fig.13 Manual checkpoints 206a and 206b are shown and displayed as sub-menus of the manual checkpoint button 202b. Fig.14 Instrument checkpoints 207-c are shown, displayed as a submenu of the manual checkpoint button 202c.

[0063] Returning to step 134, if the user selects button 202e, dynamic checkpoint selection is enabled at step 146, such as Fig.16 and Fig.17 The dynamic checkpoint selection includes a video stream 208a of the entire surgery or a portion thereof, and playback controls 208b that allow the user to jump to a desired point and select a dynamic checkpoint via button 208c. The video stream 208a may be displayed in any manner, such as a picture-in-picture mode or overlaid on the video feed of the camera 51. At step 148, the user may also select a previously saved dynamic checkpoint.

[0064] Once the user selects a checkpoint, checkpoint navigation is enabled at step 144 . Fig.18 The situation when one of the checkpoints 207a to 207c or any other checkpoint described above is selected is shown. The selected checkpoint (i.e., checkpoint 207a) is enlarged into a window 207d and a navigation button 207e and a back button 207f for reverting to the checkpoint selection. At step 152, the controller 21a checks which of the buttons 207e or 207f is pressed. Pressing button 207f causes the enlarged window 207d to be removed at step 154 ​​and the GUI 200 to be returned to step 142, in which the available checkpoints for a given submenu (i.e., manual, instrument, cycle, phase, etc.) are shown. Pressing navigation button 207e causes the input from the gimbals of the handle controllers 38a and 38b to be frozen at step 156 in preparation for navigating to the selected checkpoint. Each of the handle controllers 38a and 38b includes a gimbal assembly (not shown) that allows the handle controllers 38a and 38b to move and rotate about three axes (x, y, z). Details of the handle controllers 38a and 38b, including gimbal operation, are provided in U.S. Patent Application Publication No. 2020 / 0315729, entitled “Control Arm Assembly for Robotic Surgical System,” filed on November 30, 2018, the entire contents of which are incorporated herein by reference.

[0065] At step 158, the controller 21a checks whether the user has chosen to navigate backward or forward to the selected checkpoint. If the user has chosen to navigate forward, then at step 160, the GUI 200 clears the checkpoint video and / or thumbnail. The controller 21a also loads coordinates or other data (e.g., kinematics) from the memory 22a to place the robotic arms 40a to 40d and their corresponding instruments 50 and / or cameras 51 in the position and configuration of the selected checkpoint. In addition to controlling the robotic arms 40a to 40d, a separate IDU 52 is also actuated to reach the desired checkpoint. After the camera 51 is removed due to replacement of an instrument or cleaning, forward navigation can be used during the reinsertion of the instrument 50 and / or camera 51. This allows the system 10 to quickly resume operation after an interruption caused by removing the instrument 50 and / or camera 51 from the surgical site.

[0066] At step 162, the controller 21a verifies whether the user is holding the handle controller 38a and 38b. If one of the handle controllers 38a or 38b is moving, then at step 164, the controller 21a displays a prompt 209 to the user to release the handle controllers 38a and 38b. At step 166, the controller 21a automatically moves the robotic arms 40a to 40d and the associated instruments 50 / camera 51 to the loading coordinates of the checkpoint. At step 168, the controller 21a checks whether there are any other available checkpoints to be loaded based on the selected forward navigation direction. At step 170, the controller 21a continues to load subsequent checkpoints until the desired checkpoint or final checkpoint is reached, i.e., the camera 51 returns to the position it was in before being removed from the patient. During navigation, the video feed 210a of the camera 51 shows a message 210b that navigation is in progress, such as Fig. 20 In addition, the checkpoint to which the system 10 is being restored is shown in thumbnail 210c. This provides a visual confirmation to the user that the system 10 has reached the checkpoint, i.e., once feed 210a matches thumbnail 210c.

[0067] Returning to step 158, if the user selects to navigate backward, the GUI 200 clears the checkpoint video and / or thumbnail at step 172, and the controller 21a loads coordinates or other data (e.g., kinematics) from the memory 22a to place the robotic arms 40a to 40d and their corresponding instruments 50 and / or cameras 51 into the selected checkpoint location and configuration. Backward navigation can be used to remove instruments 50 and / or cameras 51 to replace instruments or clean cameras 51.

[0068] At step 174, the controller 21a verifies whether the user is holding the handle controllers 38a and 38b. If one of the handle controllers 38a or 38b is moving, then at step 176, the controller 21a displays a prompt 209 to the user to release the handle controllers 38a and 38b. At step 178, the controller 21a automatically moves the robotic arms 40a to 40d and the associated instruments 50 / cameras 51 to the loading coordinates of the checkpoint. At step 180, the controller 21a checks whether there are any other available checkpoints to be loaded based on the selected forward navigation direction. At step 182, the controller 21a continues to load subsequent checkpoints until the desired checkpoint or final checkpoint is reached, that is, the camera 51 returns to the initial position before being inserted into the patient. In another embodiment, the controller 21a can disable input from the handle controllers 38a and 38b during the movement of the robotic arms 40a to 40d to the checkpoint position.

[0069] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A surgical robot system, comprising: A surgical robot arm comprising a surgical device and an instrument drive unit configured to actuate the surgical device; a surgeon console configured to receive user input to control at least one of the surgical robotic arm or the surgical device; as well as A control tower connected to the surgical robot arm and the surgeon's console, the control tower comprising: storage device; and A controller configured to: storing a plurality of checkpoints on the storage device, each of the plurality of checkpoints comprising image data of a surgical site and position data corresponding to a previous position of the surgical robotic arm during the surgical procedure; outputting a graphical user interface (GUI) on a display screen, the GUI being configured to display image data of at least one checkpoint of the plurality of checkpoints and receive a user input selecting the at least one checkpoint; and Based on the selected at least one checkpoint, the surgical robotic arm is controlled to move the surgical robotic arm to the previous position.

2. The surgical robot system according to claim 1, wherein: The controller is further configured to store at least one checkpoint of the plurality of checkpoints in response to activation of the surgical device.

3. The surgical robot system according to claim 1, wherein: The controller is further configured to store at least one checkpoint of the plurality of checkpoints in response to a user command.

4. The surgical robot system according to claim 1, wherein: The controller is further configured to store at least one checkpoint of the plurality of checkpoints based on an elapsed time of the surgical procedure.

5. The surgical robot system according to claim 1, wherein: The controller is further configured to detect a stage of the surgical procedure.

6. The surgical robot system according to claim 5, wherein: The controller is further configured to store at least one checkpoint of the plurality of checkpoints based on the detected stage of the surgical procedure.

7. The surgical robot system according to claim 1, wherein: The surgical device is a surgical robotic instrument.

8. The surgical robot system according to claim 1, wherein: The surgical device is a laparoscopic camera.

9. The surgical robot system according to claim 8, wherein: At least one checkpoint of the plurality of checkpoints includes position data corresponding to the laparoscopic camera external to the patient.

10. A method for restoring a position state of a surgical robot system, the method comprising: storing a plurality of checkpoints on a storage device, the plurality of checkpoints including image data of a surgical site and position data corresponding to previous positions of a laparoscopic camera held by a surgical robot arm during the surgical procedure, wherein a first checkpoint of the plurality of checkpoints corresponds to a position of the laparoscopic camera outside the patient, and a second checkpoint of the plurality of checkpoints corresponds to a position of the laparoscopic camera inside the patient; outputting a graphical user interface (GUI) configured to display the image data of the plurality of inspection points on a display screen; receiving, through the GUI, a first user input selecting at least one of the first checkpoint or the second checkpoint; and The surgical robotic arm is controlled based on the selection of at least one of the first checkpoint or the second checkpoint to move the surgical robotic arm to the previous position.

11. The method of claim 10, further comprising storing at least one of the first checkpoint or the second checkpoint in response to activation of the laparoscopic camera.

12. The method of claim 10, further comprising storing at least one of the first checkpoint or the second checkpoint in response to a user command.

13. The method of claim 10, further comprising storing at least one of the first checkpoint or the second checkpoint based on an elapsed time of the surgical procedure.

14. The method according to claim 10, further comprising: Detect the stage of the surgical procedure; as well as At least one of the first checkpoint or the second checkpoint is stored based on the detected stage of the surgical procedure.

15. A method for restoring a position state of a surgical robot system, the method comprising: storing a plurality of checkpoints on a storage device, each of the plurality of checkpoints comprising image data of a surgical site and position data corresponding to a previous position of a surgical robotic arm during the surgical procedure; outputting a graphical user interface (GUI) configured to display image data of at least one of the plurality of checkpoints on a display screen; receiving, via the GUI, user input selecting the at least one checkpoint; as well as Based on the selected at least one checkpoint, the surgical robotic arm is controlled to move the surgical robotic arm to the previous position. 16 . The method of claim 15 , further comprising storing at least one of the plurality of checkpoints in response to activation of a surgical device held by the surgical robotic arm.

17. The method of claim 15, further comprising storing at least one of the plurality of checkpoints in response to a user command.

18. The method of claim 15, further comprising storing at least one of the plurality of checkpoints based on an elapsed time of the surgical procedure.

19. The method of claim 15, further comprising detecting a stage of the surgical procedure.

20. The method of claim 19, further comprising storing at least one of the plurality of checkpoints based on the detected phase of the surgical procedure.

Images